Try to invoke main class org.jetbrains.java.decompiler.main.decompiler.ConsoleDecompiler
manually
java -cp java-decompiler.jar org.jetbrains.java.decompiler.main.decompiler.ConsoleDecompiler myclassfile.class .
Also you can find more about internal usage of Fernflower inside IDEA by query in source code
https://github.com/JetBrains/intellij-community/search?utf8=%E2%9C%93&q=Fernflower