A coworker of mine found the memory leak.
If you disable these two checkboxes your problem is fixed.
Tools -> options -> Text editor -> Advanced
Disable the following:
-
Enable pull diagnostics (experimental, requires restart)
-
Enable razor pull diagnostics (experimental, requires restart)