Why must fixed size buffers (arrays) be declared unsafe?

Because a “fixed buffer” is not a real array. It is a custom value type, about the only way to generate one in the C# language that I know. There is no way for the CLR to verify that indexing of the array is done in a safe way. The code is not verifiable either. The most graphic demonstration of this:

using System;

class Program {
    static unsafe void Main(string[] args) {
        var buf = new Buffer72();
        Console.WriteLine(buf.bs[8]);
        Console.ReadLine();
    }
}
public struct Buffer72 {
    public unsafe fixed byte bs[7];
}

You can arbitrarily access the stack frame in this example. The standard buffer overflow injection technique would be available to malicious code to patch the function return address and force your code to jump to an arbitrary location.

Yes, that’s quite unsafe.