It turns out that the KeySize for X509SecurityKey needs to be at least 1024 in length for verifying. This is not obvious from the exception, since it is hidden with the [PII is hidden] filter.
Adding the following line made the exception text a lot more useful (add to ConfigureServices method in Startup.cs):
IdentityModelEventSource.ShowPII = true;
The new exception text:
‘System.ArgumentOutOfRangeException: IDX10631: The ‘Microsoft.IdentityModel.Tokens.X509SecurityKey’ for verifying cannot be smaller than ‘1024’ bits. KeySize: ‘512’.
Increasing the length of the assymetric key to 1024 solved the problem.