Update 1:
I created a new project without running the Swift 3 conversion, imported my Swift 3 files, but the build time remains the same.
Update 2:
I’ve tried SWIFT_WHOLE_MODULE_OPTIMIZATION = YES, but as soon as you make changes to more than 1 file, the incremental build fails and it triggers a re-build which lasts for more than 4-5 minutes.
Update 3:
I’ve now re-written my entire code base from Swift 3 to Swift 2.3. It didn’t make any difference, the problem lies with the Xcode 8 compiler.
Update 4:
I can confirm that unchecking these two 
will alleviate the pain for a while, the Xcode 8 bug does seem to be tied to how it checks dependencies between files.
Update 5:
I’ve converted my code base to Swift 3 from Swift 2.3 since Xcode 8.2 beta requires it, the beta should include a fix for “Xcode will not rebuild an entire target when only small changes have occurred. (28892475)”. Sad to say, they haven’t fixed the bug and my compile times are exactly the same with Xcode 8.2 Beta.
Original post:
I don’t have enough reputation to comment, but I still wanted to share some resources. I’ve been stuck in this misery for days, upgrading to Swift 3 has been a complete disaster.
I’m using this to track slow files, even though just like you, that’s not my problem. Something else in xcode is taking literally 4 minutes to complete:
https://github.com/irskep/swift_compile_times_parser
https://github.com/RobertGummesson/BuildTimeAnalyzer-for-Xcode
I’ve also made sure I don’t have any lazy vars or closures that swift doesn’t like. Don’t use the + operator when concatenating strings, etc.
see this.
I’ll update this answer if I find anything, it’s just about impossible to be productive with Swift 3 ATM.