micro benchmarking with openjdks jmh

Micro benchmarking in Java

What is micro benchmarking?

It’s for benchmarking very small parts of logic. Small like e.g. arraylist.add or string.split.

Why do we have a problem with micro benchmarking?

Often you encounter some blog post where someone claims that x is faster than y. It is faster in the execution the author has done, but there are many circumstances to take into consideration before you really can say that x is faster than y.
Especially for Java, where we have a Virtual Machine, JIT, GC, etc, which makes getting scientific micro benchmarking results rather difficult.

Solution – use JMH framework for micro benchmarking

There are some tools out there, which make it easy to write micro benchmarkable code. I will here focus on jmh. This tool is developed by the openjdk team. So we can expect that they know the VM internals and its pitfalls.

As described on their main page just grab the maven archetype “jmh-java-benchmark-archetype” and you are good to go. An archetype is like a blueprint for your maven project.

You can integrate it into eclipse like any other maven project. Just be aware that validation is only done if you build the project with maven. The validation is very important. Otherwise you start the application out of your IDE and wonder why it doesn’t work like expected.

micro benchmarking

After the archetype is generated you have a class named MyBenchmark with a method testMethod. It looks like:

 

Let’s start micro benchmarking

To start you can just fill the provided testMethod with code to benchmark. For example:

You may ask why did I change the method signature? It’s important to return the value otherwise we have dead code, which will be removed by the compiler.

Now build it

Now execute “mvn clean install” to build it in maven, which additionaly does some validation regarding jmh. So if something goes wrong let maven build.

Or you can just press save in eclipse. 😛

Execute it

Use either “java -jar target/benchmarks.jar”.

Or execute the main method org.openjdk.jmh.Main

Now the benchmark is executed in the default configuration. This takes pretty long, because many warmup iterations and many measurement iterations are done. But with that setup you are ready to go and do some simple benchmarking.

Enhance your benchmark

There are several ways to enhance your benchmarks. First of all I recommend the samples provided by jmh. They are really cool and explain themselves in their javadoc. Yet I will go into details of some points.

Make your own configuration of how to micro benchmark

20 warmup iterations, 20 measured iterations. Man that takes time. I don’t need it to be that exact. So let’s do our own configuration. Just create our own micro benchmark configuration using a main method. Let maven build again and we are ready to go 😉

 

Everything you see in the first lines of the console can be changed in the main method. For example:

Here we use AverageTime instead of Throughput. So we get the execution time of the method.
It is important to change timeunit too, otherwise you may get 0 seconds everywhere, because your timeunit is too small.

But imo throughput mode is better for micro benchmarking, because it often goes down to nanoseconds, which is just is too fast to measure right.

Add another method for micro benchmarking

Just add another method and you can directly compare two different implementations. It’s important that you often have to rebuild with maven!

Don’t forget in Throughput mode higher is better.

Now we can conclude in this micro benchmark that splitting with apache commons for two chars separator is better than Java split. But for one char separator Javas core implementation is similar in performance.

Code for above benchmark:

 

This entry was posted in Tooling and tagged , , , , , . Bookmark the permalink.

2 Responses to micro benchmarking with openjdks jmh

  1. Nasim says:

    I’m working on a maven project. The scenario is something like below…

    class Test {

    public void applyAll() {
    ………………..
    ………………..
    Collection applicableUpdates = SomeOtherClass.getApplicableUpdates();
    doUpdate(applicableUpdates);

    }

    @Benchmark
    public void apply(Migratable m) {
    ………………..
    DB operations
    ………………..
    }

    private void doUpdate(Collection applicableUpdates) throws Exception {
    for (Migratable m : applicableUpdates) {
    try {

    apply(m);

    } catch (Exception e) {
    logger.error(“Falid to apply migration {}” + m, e);
    throw e;
    }
    }
    }
    }

    I need to benchmark apply(Migratable m) method.
    How can I do that?

    • keiki says:

      JMH is not suited in my opinion, because..

      * databases are not micro and can have unwanted effects
      * JMH is focused for micro benchmarks. So below a few milliseconds in my opinion.
      * You want a separate project for doing micro benchmarking.

Leave a Reply

Your email address will not be published. Required fields are marked *