Actually I found a solution for now, see diff below:

It sounds like this may be bug of TestServer() class and how it is hosting the application during the test run.
Here is the line of code in case you cannot read above in image
.AddApplicationPart(Assembly.Load(new AssemblyName("WebApiToReproduceBug.Controllers")));