You can achieve your desired result. You need to switch to NewtonsoftJson serialization (from package Microsoft.AspNetCore.Mvc.NewtonsoftJson)
Call this in Startup.cs in the ConfigureServices method:
services.AddControllers().AddNewtonsoftJson();
After this, your constructor will be called by deserialization.
Extra info: I am using ASP Net Core 3.1
Later Edit: I wanted to give more info on this, as it seems that this can also be achieved by using System.Text.Json, although custom implementation is necessary. The answer from jawa states that Deserializing to immutable classes and structs can be achieved with System.Text.Json, by creating a custom converter (inherit from JsonConverter) and registering it to the converters collection (JsonSerializerOptions.Converters) like so:
public class ImmutablePointConverter : JsonConverter<ImmutablePoint>
{
...
}
and then…
var serializeOptions = new JsonSerializerOptions();
serializeOptions.Converters.Add(new ImmutablePointConverter());
serializeOptions.WriteIndented = true;