I had the same problem. From a related GitHub Issue, I learned that you can extend the inactivity timeout.
Set this Karma config option in your gruntfile or karma config file:
browserNoActivityTimeout: 100000
I set it to 100 seconds and my tests ran successfully. I don’t know what’s causing the delay.