After a lot of digging (I had this issue for a long time before posting) I found the issue. One of the NuGet package sources in my list was down. If I remove that from the list, everything goes smoothly. With it in, it bogs down even if no packages in my packages.config are related to that source.