Unlike adb shell the adb exec-out command doesn’t use pty which mangles the binary output. So you can do
adb exec-out screencap -p > test.png
https://android.googlesource.com/platform/system/core/+/5d9d434efadf1c535c7fea634d5306e18c68ef1f
Note that if you are using this technique for a command that produces output on STDERR, you should redirect it to /dev/null, otherwise adb will include STDERR in its STDOUT corrupting your output. For example, if you are trying to backup and compress a directory:
adb exec-out "tar -zcf - /system 2>/dev/null" > system.tar.gz