How can I ssh directly to a particular directory?

You can do the following:

ssh -t xxx.xxx.xxx.xxx "cd /directory_wanted ; bash --login"

This way, you will get a login shell right on the directory_wanted.

Explanation

-t Force pseudo-terminal allocation. This can be used to execute arbitrary screen-based programs on a remote machine, which can be very useful, e.g. when implementing menu services.

Multiple -t options force tty allocation, even if ssh has no local tty.

• If you don’t use -t then no prompt will appear.
• If you don’t add ; bash then the connection will get closed and return control to your local machine
• If you don’t add bash --login then it will not use your configs because its not a login shell