You can use ssh-agent to remember your passphrase (Gnome automatically runs this for you, normally…).
$ ssh-agent bash
$ ssh-add
Enter passphrase for /home/elyobo/.ssh/id_rsa:
Identity added: /home/elyobo/.ssh/id_rsa (/home/elyobo/.ssh/id_rsa)
From now on, from within the terminal that you run this, your pass phrase will be remembered.
Ideally you’d get it working automatically, so all shells running within gnome would work; check out Gnome Keyring.