You could encrypt the private key with a symmetric key based on the users password. Simply store an additional salt and perform the password “hash” to get a separate key. Then use that as key for encrypting the private key. Note that it is required to use a Password Based Key Derivation Function (PBKDF) such as PBKDF2, bcrypt or scrypt to create a secure password hash (given the normal security level of a password).
If the user is not online at the time that a signature needs to be generated, then you should indeed protect the passwords in a sense that only you / our backoffice can decrypt the keys. You can use some user ID + your own secret key to calculate an encryption/decryption key. You may even want to generate a separate RSA key pair to perform hybrid encryption decryption.
Storing private keys on behalf of users is a very dangerous practice. There are a lot of ways for the private key to become exposed (e.g. side channel attacks). To do it professionally you should really be using an HSM somewhere in the process. If this is for any serious data, please consult a professional and a lawyer.