Micro benchmarking in Java

What is micro benchmarking?

It’s for benchmarking very small parts of logic. Small like e.g. arraylist.add or string.split.

Why do we have a problem with micro benchmarking?

Often you encounter some blog post where someone claims that x is faster than y. It is faster in the execution the author has done, but there are many circumstances to take into consideration before you really can say that x is faster than y.
Especially for Java, where we have a Virtual Machine, JIT, GC, etc, which makes getting scientific micro benchmarking results rather difficult.

Solution – use JMH framework for micro benchmarking

There are some tools out there, which make it easy to write micro benchmarkable code. I will here focus on jmh. This tool is developed by the openjdk team. So we can expect that they know the VM internals and its pitfalls.

As described on their main page just grab the maven archetype “jmh-java-benchmark-archetype” and you are good to go. An archetype is like a blueprint for your maven project.

You can integrate it into eclipse like any other maven project. Just be aware that validation is only done if you build the project with maven. The validation is very important. Otherwise you start the application out of your IDE and wonder why it doesn’t work like expected.

After the archetype is generated you have a class named MyBenchmark with a method testMethod. It looks like:

package sandbox;

import org.openjdk.jmh.annotations.Benchmark;

public class MyBenchmark {

    @Benchmark
    public void testMethod() {
        // place your benchmarked code here
    }

}

Let’s start micro benchmarking

To start you can just fill the provided testMethod with code to benchmark. For example:

    @Benchmark
    public String[] testMethod() {
        return "1,2,3,4,5,6,7,8,9,10".split(",");
    }

You may ask why did I change the method signature? It’s important to return the value otherwise we have dead code, which will be removed by the compiler.

Now build it

Now execute “mvn clean install” to build it in maven, which additionaly does some validation regarding jmh. So if something goes wrong let maven build.

Or you can just press save in eclipse. 😛

Execute it

Use either “java -jar target/benchmarks.jar”.

Or execute the main method org.openjdk.jmh.Main

Now the benchmark is executed in the default configuration. This takes pretty long, because many warmup iterations and many measurement iterations are done. But with that setup you are ready to go and do some simple benchmarking.

Enhance your benchmark

There are several ways to enhance your benchmarks. First of all I recommend the samples provided by jmh. They are really cool and explain themselves in their javadoc. Yet I will go into details of some points.

Make your own configuration of how to micro benchmark

20 warmup iterations, 20 measured iterations. Man that takes time. I don’t need it to be that exact. So let’s do our own configuration. Just create our own micro benchmark configuration using a main method. Let maven build again and we are ready to go 😉

    public static void main(String[] args) throws RunnerException {
        Options opt = new OptionsBuilder()
                .include(MyBenchmark.class.getSimpleName())
                .warmupIterations(5)
                .measurementIterations(5)
                .forks(1)
                .build();

        new Runner(opt).run();
    }

Everything you see in the first lines of the console can be changed in the main method. For example:

    public static void main(String[] args) throws RunnerException {
        Options opt = new OptionsBuilder()
                .include(MyBenchmark.class.getSimpleName())
                .warmupIterations(5)
                .mode(Mode.AverageTime)
                .timeUnit(TimeUnit.NANOSECONDS)
                .measurementIterations(5)
                .forks(1)
                .build();

        new Runner(opt).run();
    }

Here we use AverageTime instead of Throughput. So we get the execution time of the method.
It is important to change timeunit too, otherwise you may get 0 seconds everywhere, because your timeunit is too small.

But imo throughput mode is better for micro benchmarking, because it often goes down to nanoseconds, which is just is too fast to measure right.

Add another method for micro benchmarking

Just add another method and you can directly compare two different implementations. It’s important that you often have to rebuild with maven!

# Run complete. Total time: 00:00:49

Benchmark                                              Mode  Samples        Score  Score error  Units
s.StringSplitBenchmark.testApacheCommonsSingleChar    thrpt        5  4310855.795   101878.883  ops/s
s.StringSplitBenchmark.testApacheCommonsTwoChars      thrpt        5  3230103.069   261743.521  ops/s
s.StringSplitBenchmark.testJavaStandardSingleChar     thrpt        5  4438152.511   195093.492  ops/s
s.StringSplitBenchmark.testJavaStandardTwoChars       thrpt        5  1690683.608    83516.882  ops/s

Don’t forget in Throughput mode higher is better.

Now we can conclude in this micro benchmark that splitting with apache commons for two chars separator is better than Java split. But for one char separator Javas core implementation is similar in performance.

Code for above benchmark:

package sandbox;

import org.apache.commons.lang3.StringUtils;
import org.openjdk.jmh.annotations.Benchmark;
import org.openjdk.jmh.runner.Runner;meone claims that x is faster than y. It is faster in the execution the author has done, but there are many circumstances to take into consideration before you really can say that x is faster than y.
Especially for Java, where we have a Virtual Machine, JIT, GC, etc, which makes getting scientific micro benchmarking results rather diffic
import org.openjdk.jmh.runner.RunnerException;
import org.openjdk.jmh.runner.options.Options;
import org.openjdk.jmh.runner.options.OptionsBuilder;

public class StringSplitBenchmark {

    @Benchmark
    public String[] testJavaStandardSingleChar() {
        return "1,2,3,4,5,6,7,8,9,10".split(",");
    }
    
    @Benchmark
    public String[] testApacheCommonsSingleChar() {
        return StringUtils.split("1,2,3,4,5,6,7,8,9,10",',');
    }
    
    @Benchmark
    public String[] testJavaStandardTwoChars() {
        return "1,.2,.3,.4,.5,.6,.7,.8,.9,.10".split(",\\.");
    }
    
    @Benchmark
    public String[] testApacheCommonsTwoChars() {
        return StringUtils.split("1,.2,.3,.4,.5,.6,.7,.8,.9,.10",",.");
    }
    
    public static void main(String[] args) throws RunnerException {
        Options opt = new OptionsBuilder()
                .include(StringSplitBenchmark.class.getSimpleName())
                .warmupIterations(5)
                .measurementIterations(5)
                .forks(1)
                .build();

        new Runner(opt).run();
    }
}