ASP.NET Core can appear to use more memory than it should because it’s configured to use Server GC by default (as opposed to Workstation GC). For a discussion of a similar concern see here. In theory, the application should be able to reduce its memory footprint when your server faces a memory pressure.
More on this topic in this MSDN doc and Github CoreCLR doc.
To see if it’s indeed Server GC that causes the additional memory consumption, you can set GC strategy to workstation in your csproj file:
<PropertyGroup>
<ServerGarbageCollection>false</ServerGarbageCollection>
</PropertyGroup>