I experienced a similar issue but the problem, in my case, is not related to speed but something different.
Specifically, if you force your iOS Simulator to run under Rosetta you’ll see that it works as expected. Not sure why, but this is what I’m experiencing. Always reproducible with 1.22.5 (stable).
To run in Rosetta, right click on Xcode and choose “Show Package Contents”, from there you navigate to “Contents > Developer > Applications,” There you’ll find the Simulator app. If you right click on it and choose “Get Info”, you’ll find an option to run it using Rosetta.
Update 14/9/2021: Xcode 12.5.1 no longer needs the above workaround. Scrolling is working fine out-of-the-box.