As previously mentioned, rolling back the changeset rolls it back in its entirety.
Installing Team Foundation Power Tools (which I think pretty much everyone should install) gives you another option: If you want to rollback a single file, you can right click a single file in the Source Control Explorer and select Rollback…