How to solve “sign_and_send_pubkey: signing failed: agent refused operation”?

Run ssh-add on the client machine, that will add the SSH key to the agent.

Confirm with ssh-add -l (again on the client) that it was indeed added.