Found this after I had posted the question:
https://pip.pypa.io/en/stable/topics/configuration/#configuration-files
~/.pip/pip.conf
You will need to set:
~/.virtualenvs/env1/pip.conf
~/.virtualenvs/env2/pip.conf
It will still inherit from:
~/.pip/pip.conf
But will allow it to be overwritten for each environment.