The keys need to be read-writable only by you:
chmod 600 ~/.ssh/id_rsa
Alternatively, the keys can be only readable by you (this also blocks your write access):
chmod 400 ~/.ssh/id_rsa
600 appears to be better in most cases, because you don’t need to change file permissions later to edit it. (See the comments for more nuances)
The relevant portion from the manpage (
~/.ssh/id_rsa Contains the private key for authentication. These files contain sensitive data and should be readable by the user but not accessible by others (read/write/execute). ssh will simply ignore a private key file if it is accessible by others. It is possible to specify a passphrase when generating the key which will be used to encrypt the sensitive part of this file using 3DES. ~/.ssh/identity.pub ~/.ssh/id_dsa.pub ~/.ssh/id_ecdsa.pub ~/.ssh/id_rsa.pub Contains the public key for authentication. These files are not sensitive and can (but need not) be readable by anyone.