Use the following:
:set mouse=a
to turn on xterm style mousing in all modes. This will allow you to do what you want. Keep in mind if the vim is remote via ssh, you’ll need X11 forwarding turned on for the selection to make it to your local clipboard.