The CLR rules of casting specify that this is possible. The C# rules say it is not possible. The C# team consciously decided that they would tolerate this deviation from the spec for various reasons.
Why does the CLR allow this? Probably because they can conveniently implement it. byte and sbyte have the same binary representation so you can “treat” a byte[] as an sbyte[] without violating memory safety.
The same trick works for other primitive types with the same memory layout.