You can edit the .git/config file and remove the section associated with the remote you want to remove.
The lines you want to remove probably look roughly like this:
[svn-remote "svn"]
url = url-of-svn-repository/trunk
fetch = :refs/remotes/git-svn
This will remove the remote from Git, but you will still have the relationship between the Git commits and the SVN commits stored in .git/svn. You can remove this whole directory if you want to get rid of all your SVN remotes. If you want to get rid of just one SVN remote and keep others, you will need to look at the directory structure and only remove the ones associated with the remote you are removing.