Go to project settings, then Editor > Add Build Setting > Add User-Defined Setting, and add the following:
SWIFT_WHOLE_MODULE_OPTIMIZATION = YES
Adding this flag dropped our clean-build compile times from 7 mins to 65s for a 40KLOC swift project, miraculously. Also can confirm 2 friends have seen similar improvements on enterprise projects.
I can only assume this is some kind of bug in Xcode 8.0