Convention is one of the reasons – most of the time, with effective abstraction, you only care about the interface and want to have it easy just looking at the headers.
It’s not the only reason though. If your project is organised in modules, you most likely have to include some headers in different modules, and you want your include directory to be cleaned of other “noise” files in there.
Also, if you plan on redistributing your module, you probably want to hide implementation details. So you only supply headers and binaries – and distributing headers from a single folder with nothing else in it is simpler.
There’s also an alternative which I actually prefer – public headers go in a separate folder (these contain the minimum interface – no implementation details are visible whatsoever), and private headers and implementation files are separate (possibly, but not necessarily, in separate folders).