For iOS 10.x and greater to reduce AVPlayer start delay I set:
avplayer.automaticallyWaitsToMinimizeStalling = false;
and that seemed to fix it for me. This could have other consequences, but I haven’t hit those yet.
I got the idea for it from:
https://stackoverflow.com/a/50598525/9620547