After some additional investigation I found this thread
Unchecking Enable Diagnostic Tools while debugging in Tools → Options → Debugging → General did the trick!
Unchecking Enable JavaScript debugging for ASP.NET in
Tools → Options → Debugging → General
makes a huge difference in performance.