You need to add a T : struct constraint:
public static Nullable<T> CoalesceMax<T>
(Nullable<T> a, Nullable<T> b) where T : struct, IComparable
Otherwise C# will try to work out what Nullable<T> means, and realise that it doesn’t already have the constraint required by Nullable<T> itself. In other words, you could try to call:
CoalesceMax<string>(...)
which wouldn’t make sense, as Nullable<string> isn’t valid.