Often, the largest thing you can do is to control your inclusion of header files.
Including “extra” header files in source code dramatically slows down the compilation. This also tends to increase the time required for dependency checking.
Also, using forward declaration instead of having headers include other headers can dramatically reduce the number of dependencies, and help all of your timings.