my _vimrc/_gvimrc is stored at C:/Users/<ME>/_vimrc and is working fine.
It’s generally a good idea to keep personal settings separate from installation files.
To get more information about the search paths for your configuration files, type :he vimrc-intro.
And be careful: the docs say
For MS-DOS and MS-Windows you can use one of these:
$HOME/_vimrc
$VIM/_vimrc
While this is absolutely true, it could be a bit surprising that $VIM does not expand to e.g. C:/Program Files/vim/<your_version> (this is what $VIMRUNTIME holds) but only to C:/Program Files/vim/.
This said, C:/Program Files/vim/_vimrc should be read during startup.