How to run JMH from inside JUnit tests?

I’ve been running JMH inside my existing Maven project using JUnit with no apparent ill effects. I cannot answer why the authors recommend doing things differently. I have not observed a difference in results. JMH launches a separate JVM to run benchmarks to isolate them. Here is what I do:

  • Add the JMH dependencies to your POM:


    Note that I’ve placed them in scope test.

    In Eclipse, you may need to configure the annotation processor manually. NetBeans handles this automatically.

  • Create your JUnit and JMH class. I’ve chosen to combine both into a single class, but that is up to you. Notice that OptionsBuilder.include is what actually determines which benchmarks will be run from your JUnit test!

    import java.util.ArrayList;
    import java.util.List;
    import java.util.Random;
    import java.util.concurrent.TimeUnit;
    import org.junit.Test;
    import org.openjdk.jmh.annotations.*;
    import org.openjdk.jmh.infra.Blackhole;
    import org.openjdk.jmh.runner.Runner;
    import org.openjdk.jmh.runner.options.*;
    public class TestBenchmark 
          @Test public void 
        launchBenchmark() throws Exception {
                Options opt = new OptionsBuilder()
                    // Specify which benchmarks to run. 
                    // You can be more specific if you'd like to run only one benchmark per test.
                    .include(this.getClass().getName() + ".*")
                    // Set the following options as needed
                    .mode (Mode.AverageTime)
                    //.jvmArgs("-XX:+UnlockDiagnosticVMOptions", "-XX:+PrintInlining")
                new Runner(opt).run();
        // The JMH samples are the best documentation for how to use it
        @State (Scope.Thread)
        public static class BenchmarkState
            List<Integer> list;
              @Setup (Level.Trial) public void
            initialize() {
                    Random rand = new Random();
                    list = new ArrayList<>();
                    for (int i = 0; i < 1000; i++)
                        list.add (rand.nextInt());
          @Benchmark public void 
        benchmark1 (BenchmarkState state, Blackhole bh) {
                List<Integer> list = state.list;
                for (int i = 0; i < 1000; i++)
                    bh.consume (list.get (i));
  • JMH’s annotation processor seems to not work well with compile-on-save in NetBeans. You may need to do a full Clean and Build whenever you modify the benchmarks. (Any suggestions appreciated!)

  • Run your launchBenchmark test and watch the results!

     T E S T S
    Running com.Foo
    # JMH version: 1.21
    # VM version: JDK 1.8.0_172, Java HotSpot(TM) 64-Bit Server VM, 25.172-b11
    # VM invoker: /usr/lib/jvm/java-8-jdk/jre/bin/java
    # VM options: <none>
    # Warmup: 2 iterations, 1 s each
    # Measurement: 2 iterations, 1 s each
    # Timeout: 10 min per iteration
    # Threads: 2 threads, will synchronize iterations
    # Benchmark mode: Average time, time/op
    # Benchmark: com.Foo.benchmark1
    # Run progress: 0.00% complete, ETA 00:00:04
    # Fork: 1 of 1
    # Warmup Iteration   1: 4.258 us/op
    # Warmup Iteration   2: 4.359 us/op
    Iteration   1: 4.121 us/op
    Iteration   2: 4.029 us/op
    Result "benchmark1":
      4.075 us/op
    # Run complete. Total time: 00:00:06
    REMEMBER: The numbers below are just data. To gain reusable insights, you need to follow up on
    why the numbers are the way they are. Use profilers (see -prof, -lprof), design factorial
    experiments, perform baseline and negative tests that provide experimental control, make sure
    the benchmarking environment is safe on JVM/OS/HW level, ask for reviews from the domain experts.
    Do not assume the numbers tell you what you want them to tell.
    Benchmark                                Mode  Cnt  Score   Error  Units
    Foo.benchmark1                           avgt    2  4.075          us/op
    Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.013 sec
  • even returns RunResult objects on which you can do assertions, etc.

Leave a Comment