OK, Turns out if you watch the WWDC video, they explain it:
https://developer.apple.com/videos/play/wwdc2019/416/
You need to set the Build Settings > Build Options > Build Libraries for Distribution option to Yes in your framework’s build settings, otherwise the swift compiler doesn’t generate the neccessary .swiftinterface files which are the key to future compilers being able to load your old library.
This ends up in your project.pbxproj file as:
BUILD_LIBRARY_FOR_DISTRIBUTION = YES;
After setting this flag, a framework I compiled using Xcode 11.0 (swift 5.1) was able to be used by Xcode 11.2 (swift 5.1.2) and everything appears to be working correctly.
Hopefully this question/answer will serve as a useful reference for everyone who hasn’t watched all the WWDC videos
If the error still persists go to Product > Clean Build Folder and Build again.