As you said you can disable the source control plugin going to:
- Tools / Options
- Check “Show all settings”
- Source Control / Plug-in Selection
- Set “Current source control plug-in” to “None”
Then, as Ade Miller says: Restart Visual Studio.
My Visual Studio was working really slow since the git plugging was enabled and I managed to disable it “persistently across sessions” following this steps.
Hope it helps.