Press shift while selecting with the mouse. This will make mouse selection behave as if mouse=a was not enabled.
Note: this trick also applies to “middle button paste”: if you want to paste in vim text that was selected outside, press shift while clicking the middle button. Just make sure that insert mode is activated when you do that (you may also want to :set paste to avoid unexpected effects).
OS X (mac):
hold alt/option while selecting (source)