Ah so it would seem I need to specify validateAllProperties = true
Validator.TryValidateObject(question, ctx, results, true);
Incidentally what was throwing me off was the fact I had an abstract base class with another property in it and without validateAllProperties the Validator will stop on the first error of ALL superclasses too. So you will get a validation error for each superclass (in my case 2)