It was because for some reason, this was in my csproj file:
<IISExpressAnonymousAuthentication>disabled</IISExpressAnonymousAuthentication>
Setting it to enabled fixes the issue (it can also be done from Visual Studio, select project, F4, set Anonymous Authentication in the properties grid to Enabled).