You are perhaps misunderstanding things.
byte is very much intended for “accessing memory”. You are intended to use the type when the storage is just a sequence of bytes rather than an array of characters.
Iostream types cannot be specialized with byte, since they’re designed around characters as their interface. That is, they do not think of files as sequences of bytes; they think of them as sequences of characters. Now, you can certainly read directly into a byte array by using a cast or two. But that’s not the way iostream natively thinks.
You have to make a distinction between the way iostream works and the way files work. Iostream is just one file IO library, after all; it is hardly the end-all, be-all of file APIs.
Most file APIs for reading binary data take void* rather than character arrays. std::fread/fwrite, and so forth.
That is, you should think of this, not as a problem with std::byte, but as a problem with iostream. Just another in a long line of them.