Some time ago macOS started enforcing a requirement that CA certificates can’t also be used as end-entity (eg webserver) certificates. Is it possible that iOS added this requirement between 10 and 11?
If so, the workaround is simple: you create your self-signed CA certificate, and use that certificate to issue the webserver certificate. The CA certificate (basicConstraints: CA=True) is the trust anchor that goes in your trust store; the end-entity certificate (omit basicConstraints; extendedKeyUsage=serverAuth) is presented by the web server. You’re just not allowed to use the exact same certificate for both any more.
(This should be a comment but I don’t have enough points to comment yet.)