The slowness is due to compilation indeed. But there is now the manylinux tag. Which allows the installation of pre-compiled distributions. See for example the PyPI page of numpy to see if a manylinux package is provided for your Python version.

Update (2021-06)

Since this answer received some attention lately, there are now many manylinux tags for precompiled packages (no pun intended).