I think the issue was the result of having a previous installation that wasn’t properly uninstalled. My user .cassandra directory had some default configs, that must have been for a different version, and prevented cqlsh from starting.
rm -Rf ~/.cassandra
After that, I could run cqlsh properly.
$ bin/cqlsh
Connected to Test Cluster at 127.0.0.1:9042.
[cqlsh 5.0.1 | Cassandra 2.1.2 | CQL spec 3.2.0 | Native protocol v3]
Use HELP for help.
cqlsh>