Html.ValidationSummary(false, “message”) is always showing, even on page load

If you use a developer tool in your browser to inspect the validation summary text you’ll see that it has the class validation-summary-valid when it is clear but validation-summary-errors when there are form errors.

Therefore, just create a css rule as follows;

.validation-summary-valid {
    display:none;
}

and all should be good.