Setting Whole Module Optimization to Yes is a workaround in my project.

In my case, the project has 5 modules, named here A, B, C, D and E, where the dependency is diagramed A -> B -> C -> D, and each of A to D refers E. With Xcode 6.3 and 6.3.1, it is fine to compile modules E and D, but it takes longer to compile C, and much longer to compile B and A, like 10 mins. After I change the project setting, it takes less than a min to compile all the modules. It looks faster than compilation with Xcode 6.2.
References:
- Slow test file compilation in Swift 1.2, Xcode 6.3.1
- https://devforums.apple.com/message/1128702#1128702