Using JDK8, you can set the apple.awt.application.name property to affect the Application menu name.
However, Martijn Courteaux’s warning still applies: you must do this before any AWT classes are loaded. And AWT classes will be loaded before your main() method runs if it lives in a subclass of JFrame.
Reference:
http://hg.openjdk.java.net/jdk8u/jdk8u/jdk/file/5c1d06cd7d7b/src/macosx/native/sun/osxapp/NSApplicationAWT.m#l157