Debug -> Delete All Breakpoints ( http://darrinbishop.com/blog/2010/06/sharepoint-2010-hangs-after-visual-studio-2010-f5-debugging )
After that you can use them again, but do it once. It will remove some kind of “invalid” breakpoints too and then loading symbols will be fast again. I was chasing this issue for days :(.