I would propose that the API being in the same project as your website isn’t a bad thing as long as the code is DRY*. Like you pointed out, having separate codebases is a challenge because you have to keep them in sync with every feature/bugfix you do. It’s easier to maintain if they are in the same place. As long as you keep your code DRY, this method is the clear winner.
I would offer XML and JSON from the controllers with a subdomain handled by Rails’s routing engine. When someone has picked up on the pattern of api.site.com/resource.xml and tries to access a resource that isn’t there, it’s really not a big deal. As long as your API is documented clearly and you fail/error gracefully when they try to access a resource not in your API, it should be just fine. I would try to return a message saying that resource isn’t available and a url to your api documentation. This shouldn’t be a runtime problem for any API consumers, as this should be part of discovering your API.
Just my $0.02.
*DRY = Don’t Repeat Yourself. DRY code means you don’t copy-paste or rewrite the same thing for your site and your api; you extract and call from multiple places.