Adding the -v
option to ssh when trying to log in will give a lot of debug information which might give a clue to exactly what the problem is, like for instance
debug1: Remote: No xauth program; cannot forward with spoofing.
which in my case installing xauth on the server fixed the issue.