Do you have your ssh key added to the list of agent identites ?
You can check with ssh-add -L , you should see the key your are using to connect to github :
$ ssh-add -L
ssh-rsa AAAAB3N.....0VmSiRvTzBrbU0ww== /Users/youruser/.ssh/id_rsa
If you don’t see the ssh key you use for github or a message like
The agent has no identities.
Then you should add your key with :
ssh-add ~/.ssh/id_rsa
(replace with the path to the key you use for github)
See the ssh-add doc for more info