I had the same problem but it was because my server only accept ssh login from users of “sshusers”.
On /etc/ssh/sshd_config I had the following line :
AllowGroups sshusers
In order to fix this issue, I added git to sshusers group:
$ sudo adduser git sshusers
And then it’s working.