Add an extra apostrophe ' to the MessageFormat pattern String to ensure the ' character is displayed
String text =
java.text.MessageFormat.format("You''re about to delete {0} rows.", 5);
^
An apostrophe (aka single quote) in a MessageFormat pattern starts a quoted string and is not interpreted on its own. From the javadoc
A single quote itself must be represented by doubled single quotes ” throughout a String.
The String You\\'re is equivalent to adding a backslash character to the String so the only difference will be that You\re will be produced rather than Youre. (before double quote solution '' applied)