You can surround literal strings with quotes, which for longer strings is probably easier and a bit more readable than escaping every character with a backslash:
str = String.Format("{0:MMM d 'at' m:mm"+yearStr+"}", dt);
See Custom Date and Time Format Strings in MSDN Library (search for “Literal string delimiter”).
(And did you mean h:mm instead of m:mm?)