The IdentityFile configuration parameter should be pointed at the private key which the SSH client uses to prove its identity to the remote server. (The remote server, then, should have the contents of id_rsa.pub installed in its authorized_keys file, or an equivalent location).
You should be putting the path to id_rsa, not id_rsa.pub, as an argument to IdentityFile in your ~/.ssh/config.