My final solution was a combination of the suggestions, wanting drag scrolling everywhere, so used this:
class MyApp extends StatelessWidget {
@override
Widget build(BuildContext context) {
return MaterialApp(
scrollBehavior: MaterialScrollBehavior().copyWith(
dragDevices: {PointerDeviceKind.mouse, PointerDeviceKind.touch, PointerDeviceKind.stylus, PointerDeviceKind.unknown},
),
...
Simple enough…