Installation

Direct Inclusion

1. Download nanobench.h from the release and make it available in your project.

2. Create a .cpp file, e.g. nanobench.cpp, where the bulk of nanobench is compiled.

   nanobench.cpp

   #define ANKERL_NANOBENCH_IMPLEMENT
   #include <nanobench.h>
   

3. Compile e.g. with g++ -O3 -I../include -c nanobench.cpp. This compiles the bulk of nanobench, and took 2.4 seconds on my machine. It needs to be compiled only once whenever you upgrade nanobench.

CMake Integration

nanobench can be integrated with CMake’s FetchContent or as a git submodule. Here is a full example how to this can be done:

CMakeLists.txt

cmake_minimum_required(VERSION 3.14)
set(CMAKE_CXX_STANDARD 17)

project(
    CMakeNanobenchExample
    VERSION 1.0
    LANGUAGES CXX)

include(FetchContent)

FetchContent_Declare(
    nanobench
    GIT_REPOSITORY https://github.com/martinus/nanobench.git
    GIT_TAG v4.1.0
    GIT_SHALLOW TRUE)

FetchContent_MakeAvailable(nanobench)

add_executable(MyExample my_example.cpp)
target_link_libraries(MyExample PRIVATE nanobench)

Usage

1. Create the actual benchmark code, in full_example.cpp:

   full_example.cpp

   // No ANKERL_NANOBENCH_IMPLEMENT here on purpose: this example is linked against a separately
   // compiled nanobench.cpp, which is where the implementation lives. Defining it here as well would
   // give duplicate symbols at link time. See full_example_simple.cpp for the single-file variant.
   #include <nanobench.h>
   
   #include <atomic>
   
   int main() {
       int y = 0;
       std::atomic<int> x(0);
       ankerl::nanobench::Bench().run("compare_exchange_strong", [&] {
           x.compare_exchange_strong(y, 0);
       });
   }
   

   The most important entry entry point is ankerl::nanobench::Bench. It creates a benchmarking object, optionally configures it, and then runs the code to benchmark with run().

2. Compile and link the example with

   g++ -O3 -I../include nanobench.o full_example.cpp -o full_example
   

   This takes just 0.28 seconds on my machine.

3. Run ./full_example, which gives an output like this:

   |               ns/op |                op/s |    err% |          ins/op |          cyc/op |    IPC |         bra/op |   miss% |     total | benchmark
   |--------------------:|--------------------:|--------:|----------------:|----------------:|-------:|---------------:|--------:|----------:|:----------
   |                5.63 |      177,595,338.98 |    0.0% |            3.00 |           17.98 |  0.167 |           1.00 |    0.1% |      0.00 | `compare_exchange_strong`
   

   Which renders as

   ns/op	op/s	err%	ins/op	cyc/op	IPC	bra/op	miss%	total	benchmark
   5.63	177,595,338.98	0.0%	3.00	17.98	0.167	1.00	0.1%	0.00	compare_exchange_strong

   Which means that one x.compare_exchange_strong(y, 0); call takes 5.63ns on my machine (wall-clock time), or ~178 million operations per second. Runtime fluctuates by around 0.0%, so the results are very stable. Each call required 3 instructions, which took ~18 CPU cycles. There was a single branch per call, with only 0.1% mispredicted.

Nanobench does not come with a test runner, so you can easily use it with any framework you like. In the remaining examples, I’m using doctest as a unit test framework.

Important

The five columns you always get are ns/op, op/s, err%, total and benchmark. Everything between err% and total above - ins/op, cyc/op, IPC, bra/op and miss% - comes from CPU performance counters, which nanobench can only read on Linux, through perf events. Elsewhere - and inside most containers and virtual machines, even Linux ones - those columns are silently left out, and your table looks like this instead:

|               ns/op |                op/s |    err% |     total | benchmark
|--------------------:|--------------------:|--------:|----------:|:----------
|                5.63 |      177,595,338.98 |    0.0% |      0.00 | `compare_exchange_strong`

That is not a misconfiguration of your benchmark. On Linux you may be able to get the extra columns by changing permissions through perf_event_paranoid or an ACL.

Output Columns

Each row is one benchmark. All measurements are the median over the epochs (11 by default), never a single reading, and op refers to whatever you set with unit() and batch() - by default a single call of your lambda.

Column	Platform	Meaning
ns/op	all	Wall-clock time for one operation. The header follows timeUnit(), so it can also read ms/op, us/op or ps/op.
op/s	all	Operations per second - simply the reciprocal of the time per operation.
err%	all	Median absolute percentage error of the per-epoch timings: how much the individual epochs disagreed with each other. It is a stability measure of this run, not a confidence interval, and not a comparison against any other benchmark. Above 5% the row is flagged as unstable with a :wavy_dash: marker.
ins/op	Linux only	Retired CPU instructions per operation.
cyc/op	Linux only	CPU cycles per operation.
IPC	Linux only	Instructions per cycle, i.e. ins/op divided by cyc/op. Higher is better; it says how well the operation keeps the pipeline busy.
bra/op	Linux only	Retired branch instructions per operation.
miss%	Linux only	Percentage of those branches that were mispredicted.
total	all	Total wall-clock time in seconds that this row cost to measure - the sum over all epochs of iterations times elapsed time. It is the price you paid for the measurement, not a property of the code being benchmarked. Use it to find benchmarks that make your suite slow.
benchmark	all	The name passed to run(), or the title when relative() is used.

When relative() is enabled, a leading relative column is added that compares each row against the first one.

Note

Nanobench measures time and, where available, the CPU performance counters listed above. It does not measure memory usage, allocations, or peak RSS - use a heap profiler such as heaptrack or valgrind --tool=massif for that, or count allocations yourself and report them with context().

Choosing the Columns

The full table is over 150 characters wide and wraps in most terminals. Hide what you are not reading with hideColumn(), which takes an ankerl::nanobench::Column:

ankerl::nanobench::Bench()
    .hideColumn(ankerl::nanobench::Column::instructions)
    .hideColumn(ankerl::nanobench::Column::cycles)
    .hideColumn(ankerl::nanobench::Column::ipc)
    .run("narrow", [] { /* ... */ });

Hiding only changes what is printed - the measurement still happens, and results() and the render templates are unaffected. showColumn() puts one back.

Columns that are redundant for your benchmark are worth dropping too. With unit() set to something like MFlop, MFlop/s is the number you read and ns/MFlop is the same information inverted:

bench.unit("MFlop").hideColumn(ankerl::nanobench::Column::timePerUnit);

Tip

performanceCounters(false) hides all five counter columns at once and stops measuring them, which is usually what you want if you never look at them.

Showing Context as Columns

Context variables set with context() are otherwise only reachable from a render template, which makes parameterised benchmarks hard to read on the console. contextColumn() puts one in the table:

ankerl::nanobench::Bench bench;
bench.performanceCounters(false).contextColumn("threads");

for (int threads : {1, 4, 16}) {
    bench.context("threads", std::to_string(threads))
         .run("parallel sum", [&] { /* ... */ });
}

|  threads |               ns/op |                op/s |    err% |     total | benchmark
|---------:|--------------------:|--------------------:|--------:|----------:|:----------
|        1 |               95.32 |       10,490,357.51 |    0.4% |      0.00 | `parallel sum`
|        4 |               27.11 |       36,886,742.02 |    0.7% |      0.00 | `parallel sum`
|       16 |               11.80 |       84,745,762.71 |    1.2% |      0.00 | `parallel sum`

Context columns come before the measurements, in the order you name them. A row whose context does not have the variable gets an empty cell rather than a missing column, so the table stays rectangular.

Examples

Something Fast

Let’s benchmarks how fast we can do x += x for uint64_t:

tutorial_fast_v1.cpp

#include <nanobench.h>
#include <thirdparty/doctest/doctest.h>

// NOLINTNEXTLINE
TEST_CASE("tutorial_fast_v1") {
    uint64_t x = 1;
    ankerl::nanobench::Bench().run("++x", [&]() {
        ++x;
    });
}

After 0.2ms we get this output:

|               ns/op |                op/s |    err% |     total | benchmark
|--------------------:|--------------------:|--------:|----------:|:----------
|                   - |                   - |       - |         - | :boom: `++x` (iterations overflow. Maybe your code got optimized away?)

No data there! We only get :boom: iterations overflow.. The compiler could optimize x += x away because we never used the output. Thanks to doNotOptimizeAway, this is easy to fix:

tutorial_fast_v2.cpp

#include <nanobench.h>
#include <thirdparty/doctest/doctest.h>

// NOLINTNEXTLINE
TEST_CASE("tutorial_fast_v2") {
    uint64_t x = 1;
    ankerl::nanobench::Bench().run("++x", [&]() {
        ankerl::nanobench::doNotOptimizeAway(x += 1);
    });
}

This time the benchmark runs for 2.2ms and we actually get reasonable data:

|               ns/op |                op/s |    err% |          ins/op |          cyc/op |    IPC |         bra/op |   miss% |     total | benchmark
|--------------------:|--------------------:|--------:|----------------:|----------------:|-------:|---------------:|--------:|----------:|:----------
|                0.31 |    3,192,444,232.50 |    0.0% |            1.00 |            1.00 |  0.998 |           0.00 |    0.0% |      0.00 | `++x`

It’s a very stable result. One run the op/s is 3,192 million/sec, the next time I execute it I get 3,168 million/sec. It always takes 1.00 instructions per operation on my machine, and can do this in ~1 cycle.

Something Slow

Let’s benchmark if sleeping for 100ms really takes 100ms.

tutorial_slow_v1.cpp

#include <nanobench.h>
#include <thirdparty/doctest/doctest.h>

#include <chrono>
#include <thread>

// NOLINTNEXTLINE
TEST_CASE("tutorial_slow_v1") {
    ankerl::nanobench::Bench().run("sleep 100ms, auto", [&] {
        std::this_thread::sleep_for(std::chrono::milliseconds(100));
    });
}

After 1.1 seconds I get

|               ns/op |                op/s |    err% |          ins/op |          cyc/op |    IPC |         bra/op |   miss% |     total | benchmark
|--------------------:|--------------------:|--------:|----------------:|----------------:|-------:|---------------:|--------:|----------:|:---------------------
|      100,125,753.00 |                9.99 |    0.0% |           51.00 |        7,714.00 |  0.007 |          11.00 |   90.9% |      1.10 | `sleep 100ms, auto`

So we actually take 100.125ms instead of 100ms. Next time I run it, I get 100.141. Also a very stable result. Interestingly, sleep takes 51 instructions but 7,714 cycles - so we only got 0.007 instructions per cycle. That’s extremely low, but expected of sleep. It also required 11 branches, of which 90.9% were mispredicted on average.

If the extremely slow 1.1 second is too much for you, you can manually configure the number of evaluations (epochs):

tutorial_slow_v2.cpp

#include <nanobench.h>
#include <thirdparty/doctest/doctest.h>

#include <chrono>
#include <thread>

// NOLINTNEXTLINE
TEST_CASE("tutorial_slow_v2") {
    ankerl::nanobench::Bench().epochs(3).run("sleep 100ms", [&] {
        std::this_thread::sleep_for(std::chrono::milliseconds(100));
    });
}

|               ns/op |                op/s |    err% |          ins/op |          cyc/op |    IPC |         bra/op |   miss% |     total | benchmark
|--------------------:|--------------------:|--------:|----------------:|----------------:|-------:|---------------:|--------:|----------:|:----------
|      100,099,096.00 |                9.99 |    0.0% |           51.00 |        7,182.00 |  0.007 |          11.00 |   90.9% |      0.30 | `sleep 100ms`

This time it took only 0.3 seconds, but with only 3 evaluations instead of 11. The err% will be less meaningful, but since the benchmark is so stable it doesn’t really matter.

Something Unstable

Let’s create an extreme artificial test that’s hard to benchmark, because runtime fluctuates randomly: Each iteration randomly skip between 0-254 random numbers:

tutorial_fluctuating_v1.cpp

#include <nanobench.h>
#include <thirdparty/doctest/doctest.h>

#include <random>

// NOLINTNEXTLINE
TEST_CASE("tutorial_fluctuating_v1") {
    std::random_device dev;
    std::mt19937_64 rng(dev());
    ankerl::nanobench::Bench().run("random fluctuations", [&] {
        // each run, perform a random number of rng calls
        auto iterations = rng() & UINT64_C(0xff);
        for (uint64_t i = 0; i < iterations; ++i) {
            ankerl::nanobench::doNotOptimizeAway(rng());
        }
    });
}

After 2.3ms, I get this result:

|               ns/op |                op/s |    err% |          ins/op |          cyc/op |    IPC |         bra/op |   miss% |     total | benchmark
|--------------------:|--------------------:|--------:|----------------:|----------------:|-------:|---------------:|--------:|----------:|:----------
|              334.12 |        2,992,911.53 |    6.3% |        3,486.44 |        1,068.67 |  3.262 |         287.86 |    0.7% |      0.00 | :wavy_dash: `random fluctuations` (Unstable with ~56.7 iters. Increase `minEpochIterations` to e.g. 567)

So on average each loop takes about 334.12ns, but we get a warning that the results are unstable. The median percentage error is 6.3% which is quite high,

Let’s use the suggestion and set the minimum number of iterations to 5000, and try again:

tutorial_fluctuating_v2.cpp

#include <nanobench.h>
#include <thirdparty/doctest/doctest.h>

#include <random>

// NOLINTNEXTLINE
TEST_CASE("tutorial_fluctuating_v2") {
    std::random_device dev;
    std::mt19937_64 rng(dev());
    ankerl::nanobench::Bench().minEpochIterations(5000).run(
        "random fluctuations", [&] {
            // each run, perform a random number of rng calls
            auto iterations = rng() & UINT64_C(0xff);
            for (uint64_t i = 0; i < iterations; ++i) {
                ankerl::nanobench::doNotOptimizeAway(rng());
            }
        });
}

The fluctuations are much better:

|               ns/op |                op/s |    err% |          ins/op |          cyc/op |    IPC |         bra/op |   miss% |     total | benchmark
|--------------------:|--------------------:|--------:|----------------:|----------------:|-------:|---------------:|--------:|----------:|:----------
|              277.31 |        3,606,106.48 |    0.7% |        3,531.75 |          885.18 |  3.990 |         291.59 |    0.7% |      0.00 | `random fluctuations`

The results are more stable, with only 0.7% error.

Untimed Setup

Some benchmarks consume the thing they operate on: sorting a vector leaves it sorted, so the second iteration measures sorting an already-sorted vector. setup() runs a lambda that is not measured, so the state can be restored without polluting the result:

std::vector<uint64_t> data = makeRandomData();
std::vector<uint64_t> const pristine = data;

ankerl::nanobench::Bench().setup([&] { data = pristine; })
                          .run("sort", [&] {
                              std::sort(data.begin(), data.end());
                              ankerl::nanobench::doNotOptimizeAway(data.data());
                          });

Important

The setup runs once per epoch, not once per iteration. An epoch calls your lambda many times in a row, and the setup does not run again in between. The example above is therefore not fixed by setup() alone: the first call in an epoch sorts random data, and every call after it re-sorts already-sorted data.

setup() is the right tool when the operation can be repeated as-is and only the starting state has to be established once - allocating a buffer, opening a file, warming a cache, restoring a value that the operation reads but does not destroy.

When every single call really does destroy the state, you have two honest options:

1. One iteration per epoch. epochIterations(1) makes an epoch a single call, so the setup effectively runs per iteration:

   bench.epochIterations(1).epochs(1000)
        .setup([&] { data = pristine; })
        .run("sort", [&] { std::sort(data.begin(), data.end()); });
   

   The cost is accuracy: a single call is now timed against the clock’s resolution, so this only gives useful numbers when one call takes appreciably longer than that - roughly microseconds and up. Expect a much larger err%, and use many epochs.

2. Measure the setup separately and subtract it. Benchmark just the restoration, then benchmark restoration plus operation, and take the difference. More work, but it keeps the tight measurement loop tight, and for fast operations it is the more accurate answer.

Note

Nanobench deliberately does not offer a PauseTiming()/ResumeTiming() pair inside the measurement loop. Starting and stopping the clock - and the Linux performance counters - around every iteration costs more than most operations worth benchmarking, which quietly destroys exactly the measurements it is meant to enable.

Comparing Results

To compare results, keep the ankerl::nanobench::Bench object around, enable .relative(true), and .run(…) your benchmarks. All benchmarks will be automatically compared to the first one.

As an example, I have implemented a comparison of multiple random number generators. Here several RNGs are compared to a baseline calculated from std::default_random_engine. I factored out the general benchmarking code so it’s easy to use for each of the random number generators:

example_random_number_generators.cpp (excerpt)

    }

private:
    static constexpr uint64_t rotl(uint64_t x, unsigned k) noexcept {
        return (x << k) | (x >> (64U - k));
    }

    uint64_t stateA{};
    uint64_t stateB{};
};

namespace {

// Benchmarks how fast we can get 64bit random values from Rng.
template <typename Rng>
void bench(ankerl::nanobench::Bench* bench, char const* name) {
    std::random_device dev;
    Rng rng(dev());

    bench->run(name, [&]() {
        auto r = std::uniform_int_distribution<uint64_t>{}(rng);
        ankerl::nanobench::doNotOptimizeAway(r);
    });
}

} // namespace

// NOLINTNEXTLINE
TEST_CASE("example_random_number_generators") {
    // perform a few warmup calls, and since the runtime is not always stable
    // for each generator, increase the number of epochs to get more accurate
    // numbers.
    ankerl::nanobench::Bench b;
    b.title("Random Number Generators")
        .unit("uint64_t")
        .warmup(100)
        .relative(true);
    b.performanceCounters(true);

    // sets the first one as the baseline
    bench<std::default_random_engine>(&b, "std::default_random_engine");
    bench<std::mt19937>(&b, "std::mt19937");
    bench<std::mt19937_64>(&b, "std::mt19937_64");
    bench<std::ranlux24_base>(&b, "std::ranlux24_base");
    bench<std::ranlux48_base>(&b, "std::ranlux48_base");
    bench<std::ranlux24>(&b, "std::ranlux24_base");
    bench<std::ranlux48>(&b, "std::ranlux48");
    bench<std::knuth_b>(&b, "std::knuth_b");
    bench<WyRng>(&b, "WyRng");
    bench<NasamRng>(&b, "NasamRng");
    bench<Sfc4>(&b, "Sfc4");
    bench<RomuTrio>(&b, "RomuTrio");
    bench<RomuDuo>(&b, "RomuDuo");
    bench<RomuDuoJr>(&b, "RomuDuoJr");
    bench<Orbit>(&b, "Orbit");
    bench<ankerl::nanobench::Rng>(&b, "ankerl::nanobench::Rng");
}

Runs for 60ms and prints this table:

| relative |         ns/uint64_t |          uint64_t/s |    err% |    ins/uint64_t |    cyc/uint64_t |    IPC |   bra/uint64_t |   miss% |     total | Random Number Generators
|---------:|--------------------:|--------------------:|--------:|----------------:|----------------:|-------:|---------------:|--------:|----------:|:-------------------------
|   100.0% |               35.87 |       27,881,924.28 |    2.3% |          127.80 |          114.61 |  1.115 |           9.77 |    3.7% |      0.00 | `std::default_random_engine`
|   490.3% |                7.32 |      136,699,693.21 |    0.6% |           89.55 |           23.49 |  3.812 |           9.51 |    0.1% |      0.00 | `std::mt19937`
| 1,767.4% |                2.03 |      492,786,582.33 |    0.6% |           24.38 |            6.48 |  3.761 |           1.26 |    0.6% |      0.00 | `std::mt19937_64`
|    85.2% |               42.08 |       23,764,853.03 |    0.7% |          157.07 |          134.62 |  1.167 |          19.51 |    7.6% |      0.00 | `std::ranlux24_base`
|   121.3% |               29.56 |       33,824,759.51 |    0.5% |           91.03 |           94.35 |  0.965 |          10.00 |    8.1% |      0.00 | `std::ranlux48_base`
|    17.4% |              205.67 |        4,862,080.59 |    1.2% |          709.83 |          657.10 |  1.080 |         101.79 |   16.1% |      0.00 | `std::ranlux24_base`
|     8.7% |              412.46 |        2,424,497.97 |    1.8% |        1,514.70 |        1,318.43 |  1.149 |         219.09 |   16.7% |      0.00 | `std::ranlux48`
|    59.2% |               60.60 |       16,502,276.18 |    1.9% |          253.77 |          193.39 |  1.312 |          24.93 |    1.5% |      0.00 | `std::knuth_b`
| 5,187.1% |                0.69 |    1,446,254,071.66 |    0.1% |            6.00 |            2.21 |  2.714 |           0.00 |    0.0% |      0.00 | `WyRng`
| 1,431.7% |                2.51 |      399,177,833.54 |    0.0% |           21.00 |            8.01 |  2.621 |           0.00 |    0.0% |      0.00 | `NasamRng`
| 2,629.9% |                1.36 |      733,279,957.30 |    0.1% |           13.00 |            4.36 |  2.982 |           0.00 |    0.0% |      0.00 | `Sfc4`
| 3,815.7% |                0.94 |    1,063,889,655.17 |    0.0% |           11.00 |            3.01 |  3.661 |           0.00 |    0.0% |      0.00 | `RomuTrio`
| 3,529.5% |                1.02 |      984,102,081.37 |    0.3% |            9.00 |            3.25 |  2.768 |           0.00 |    0.0% |      0.00 | `RomuDuo`
| 4,580.4% |                0.78 |    1,277,113,402.06 |    0.0% |            7.00 |            2.50 |  2.797 |           0.00 |    0.0% |      0.00 | `RomuDuoJr`
| 2,291.2% |                1.57 |      638,820,992.09 |    0.0% |           11.00 |            5.00 |  2.200 |           0.00 |    0.0% |      0.00 | `ankerl::nanobench::Rng`

It shows that ankerl::nanobench::Rng is one of the fastest RNG, and has the least amount of fluctuation. It takes only 1.57ns to generate a random uint64_t, so ~638 million calls per seconds are possible. To the left we show relative performance compared to std::default_random_engine.

Note

Here pure runtime performance is not necessarily the best benchmark. Especially the fastest RNG’s can be inlined and use instruction level parallelism to their advantage: they immediately return an old state, and while user code can already use that value, the next value is calculated in parallel. See the excellent paper at romu-random for details.

Comparing Alternatives

relative() runs one benchmark to completion, then the next, and divides the two medians. That measures the machine as much as the code. Nanobench’s own test suite records the failure: two identical workloads came out 38% apart on a CI runner, while each reported an err% of 0.5. err% is the spread within one benchmark; the comparison depends on the spread between them, and nothing in that table tells you anything about it.

compare() compares two alternatives against each other inside the same slice of time. A frequency ramp, a noisy neighbour or thermal throttling then hits both and cancels out of the ratio, and what comes back is a ratio with a confidence interval:

tutorial_compare.cpp

#include <nanobench.h>
#include <thirdparty/doctest/doctest.h>

#include <cstdint>
#include <iostream>

namespace {

// Three integer hash finalizers, to be compared against each other.
ANKERL_NANOBENCH_NO_SANITIZE("integer")
uint64_t murmurHash3Finalizer(uint64_t h) {
    h ^= h >> 33U;
    h *= UINT64_C(0xff51afd7ed558ccd);
    h ^= h >> 33U;
    h *= UINT64_C(0xc4ceb9fe1a85ec53);
    h ^= h >> 33U;
    return h;
}

ANKERL_NANOBENCH_NO_SANITIZE("integer")
uint64_t splitMix64Finalizer(uint64_t h) {
    h ^= h >> 30U;
    h *= UINT64_C(0xbf58476d1ce4e5b9);
    h ^= h >> 27U;
    h *= UINT64_C(0x94d049bb133111eb);
    h ^= h >> 31U;
    return h;
}

// One multiply and one xorshift instead of two and three.
ANKERL_NANOBENCH_NO_SANITIZE("integer")
uint64_t cheapFinalizer(uint64_t h) {
    h *= UINT64_C(0x9e3779b97f4a7c15);
    h ^= h >> 29U;
    return h;
}

} // namespace

// NOLINTNEXTLINE
TEST_CASE("tutorial_compare") {
    uint64_t x = 1;

    // 52 rounds rather than the default 11. An epoch is about a millisecond, so
    // this costs a tenth of a second and buys an interval narrow enough to act
    // on. A block is one round per alternative - two here - and a count that is
    // not a whole number of blocks gets rounded up to the next one.
    auto const cheaper = ankerl::nanobench::Bench().epochs(52).compare(
        "murmurhash3",
        [&] {
            x = murmurHash3Finalizer(x);
        },
        "cheap",
        [&] {
            x = cheapFinalizer(x);
        });

    // Two finalizers of the same shape. The interesting answer here is usually
    // that the measurement cannot tell them apart, which is a thing worth being
    // told rather than a percentage to argue over.
    auto const sameShape = ankerl::nanobench::Bench().epochs(52).compare(
        "murmurhash3",
        [&] {
            x = murmurHash3Finalizer(x);
        },
        "splitmix64",
        [&] {
            x = splitMix64Finalizer(x);
        });
    ankerl::nanobench::doNotOptimizeAway(x);

    // compare() prints the table and the verdict itself. Everything behind it
    // is available as data too, which is what a script gating a pull request
    // would look at.
    for (auto const* result : {&cheaper, &sameShape}) {
        if (result->isSignificant(1)) {
            std::cout << (*result)[1].name << " vs " << (*result)[0].name
                      << ": " << (*result)[1].relative << "x (95% CI "
                      << (*result)[1].relativeLow << " .. "
                      << (*result)[1].relativeHigh << ")" << std::endl;
        } else {
            std::cout << (*result)[1].name << " vs " << (*result)[0].name
                      << ": no difference resolved" << std::endl;
        }
    }
}

Takes about 200ms and prints an ordinary nanobench table, with the ratio to the baseline and a confidence interval for it as the first two columns:

| relative |              95% CI |               ns/op |                op/s |    err% |          ins/op |          cyc/op |    IPC |         bra/op |   miss% |     total | benchmark
|---------:|--------------------:|--------------------:|--------------------:|--------:|----------------:|----------------:|-------:|---------------:|--------:|----------:|:----------
|   100.0% |                     |                2.44 |      410,603,027.53 |    0.1% |           12.00 |           10.91 |  1.100 |           0.00 |   66.7% |      0.05 | `murmurhash3`
|   240.8% |    239.1% .. 241.0% |                1.02 |      984,999,958.21 |    0.5% |            5.00 |            4.55 |  1.099 |           0.00 |  100.0% |      0.02 | `cheap`

  Summary
    `cheap` ran 2.41x faster than `murmurhash3`
    95% CI [2.39 .. 2.41], 52 paired rounds, interleaved

The measurement columns are the ones an ordinary table has, performance counters included - a comparison collects them around every epoch anyway. Here they answer the why: cheap retires 5 instructions where murmurhash3 retires 12, which is most of the 2.4x.

More than two alternatives works the same way - the first is still the baseline, and every other row is measured against it in the same rounds. The counter columns are cut from this one to keep it narrow:

| relative |              95% CI |         ns/uint64_t |          uint64_t/s |    err% |     total | random number generators
|---------:|--------------------:|--------------------:|--------------------:|--------:|----------:|:-------------------------
|   100.0% |                     |                1.71 |      585,550,044.56 |    0.5% |      0.04 | `std::mt19937`
|    89.9% |      89.5% .. 90.2% |                1.90 |      526,080,588.04 |    0.5% |      0.04 | `std::mt19937_64`
|    45.1% |      44.8% .. 45.3% |                3.79 |      263,924,311.10 |    0.4% |      0.08 | `std::minstd_rand`
|   348.5% |    345.8% .. 349.7% |                0.49 |    2,044,545,305.41 |    0.5% |      0.01 | `nanobench::Rng`
|    86.6% |      85.5% .. 88.3% |                1.97 |      507,383,045.80 |    2.2% |      0.04 | `lcg by hand`

  Summary
    `nanobench::Rng` is fastest of 5, 3.48x ahead of `std::mt19937`
    95% CI [3.46 .. 3.50], intervals corrected for 4 comparisons, 55 paired rounds, interleaved

The interval column is the thing to read. std::mt19937_64 at 89.9% has an interval of 89.5% .. 90.2%, which excludes 100% - it really is slower. An interval that contains 100% means this experiment did not tell that row apart from the baseline, whatever its percentage says.

Note

Picking the winner out of many is a selection rather than a test: whichever came out on top is flattered by the same luck that put it there. So the summary does not claim a winner on its own - it says how far ahead of the runner-up it is, with an interval on that, and says plainly when the top two were not separated.

Reading the output

Each side’s own numbers come first, the verdict after. A ratio with no scale beside it cannot be told from the same ratio on a completely different scale, and a side that was wildly unstable is invisible in a ratio. The per-side lines are the median, nanobench’s usual err%, and the range - the same three things run() reports, so they mean what you already expect them to mean.

The interval, not the ratio, is the result. 2.41x alone is a number; 2.41x, 95% CI [2.39 .. 2.41] is a claim you can defend in a code review. If the interval were [0.9 .. 1.8] the point estimate would still say 1.3x, and it would still mean nothing.

“No difference resolved” is not “the same speed.” It says this experiment did not separate them, which is usually a reason to raise epochs() rather than a conclusion. isSignificant() is exactly the question of whether the interval excludes 1.

Watch for tied rounds. When the verdict says (38 tied at the clock's resolution), both sides measured the same time to the last tick the clock can report. That is not evidence they are equally fast, it is the clock running out of resolution, and the fix is a longer epoch via minEpochTime() - not more rounds.

Use more rounds than the default. An epoch is about a millisecond, so epochs(51) costs a tenth of a second and buys an interval narrow enough to act on. Fewer than six rounds cannot support a 95% statement at all, so compare() always runs at least eight.

How it works

Everything below is a choice, and each one is there because leaving it out changes the answer. They fall into three groups: what gets run, how a round is reduced to one number, and how those numbers become an interval.

Designing the experiment

1. A fixed iteration count, calibrated once up front. run() adapts the count as it goes; compare() does not. An iteration count that drifted between rounds would be a second thing changing while the comparison is being made.

   Calibration grows the count until an epoch’s worth of time has passed, then measures that count a second time before believing it. One reading is one interruption away from being far too high, and the count grows in steps of up to ten, so a single preempted attempt would end the search an order of magnitude early and every epoch of the comparison would come out that much shorter than it was asked to be. On a shared machine that is not a remote possibility - it is what a CI runner does several times an hour.

2. The same count for both sides. An epoch carries a fixed overhead - two clock reads and the performance counter ioctls - and what gets compared is time per iteration, so that overhead is divided by the count. Calibrating each side separately gives them slightly different counts and amortizes the overhead differently between them. That is a systematic bias in the ratio, which no amount of pairing removes: it measured 1.2% on 200µs epochs.

   Which count they share is a second question. The smallest of them is the slowest alternative’s - it needs the fewest iterations to fill an epoch - so taking it is what keeps that alternative inside maxEpochTime(). But it also makes every faster alternative run an epoch shorter than the target by however much faster it is: a 1ns operation next to a 50ns one would get a fiftieth of an epoch. So the shared count is raised until the fastest alternative clears clockResolution() * clockResolutionMultiple(), the length this library calls long enough to measure - normally far below minEpochTime(), so it only binds where the spread is wide or the clock is coarse. Once the alternatives are far enough apart the two cannot both hold, and the maximum wins: a comparison measured coarsely is better than one whose epochs are a thousand times longer than they were asked to be.

3. Interleaving. One epoch of each alternative per round, adjacent in time, rather than all of A and then all of B. Anything that affects all of them - a frequency ramp, a noisy neighbour, thermal throttling - is then common to the round and cancels out of the differences.

4. Position balanced within each block of rounds. Interleaving alone is not enough, because within a round the alternatives still run one after another: whatever goes first pays the cold cache, and whatever goes last runs on a slightly hotter core. If one alternative were always first, that cost would be attributed to it rather than to its position.

   The fix is to rotate. With N alternatives a block is N rounds, and the ordering within the block is arranged so that every alternative runs in every position exactly once. Each one then has the same mean position, and a drift that is linear over the block cancels exactly.

   With two alternatives that construction is literally ABBA:

   round 0:  A B
   round 1:  B A
   

   A is first once and second once; so is B. With three it is a cyclic Latin square - the same idea, one row per rotation:

   round 0:  A B C
   round 1:  B C A
   round 2:  C A B
   

   Read down any column: each alternative appears in it exactly once. Read across any row: each alternative appears once per round. That is what makes ABBA and the N-way case the same rule rather than two different ones - ABBA is just this square at N = 2.

5. The permutation re-randomized per block. The rotation fixes the relative order for a whole block, so a fixed starting permutation would repeat the same cycle forever and could line up with a periodic disturbance. Each block therefore starts from a fresh random permutation, which is then rotated. At N = 2 this is what picks ABBA or BAAB.

6. Rounds rounded up to whole blocks, and never fewer than six. A partial block leaves some alternative having run in the first position more often than the others, which is the imbalance the square exists to remove, so epochs() is rounded up to the next multiple of N. Six is the floor because fewer than six rounds cannot support a 95% statement at all - see the sign test below - and reporting one anyway would be inventing confidence rather than measuring it. With the intervals corrected for several comparisons the floor rises further, so a wide table is given the rounds its own interval needs.

Note

The square balances position, not carryover. In a cyclic square each alternative is always immediately preceded by the same neighbour - in the A B C block above, C never follows anything but B. If one alternative leaves the cache or the branch predictor in a state that particularly helps or hurts the next one, that effect is not balanced away, only re-randomized between blocks. Balancing it too needs a Williams design, which nanobench does not implement; the per-block reshuffle is what keeps it from becoming systematic.

Reducing a round to one number

1. The log ratio, ln(tA) - ln(tB). A speedup is multiplicative, and logs turn that into a difference, which is what every statistic below assumes. It also makes the scale symmetric: twice as fast and half as fast are the same distance from zero, where the raw ratios 2.0 and 0.5 are not. Because ln is monotonic the median commutes with it, so exponentiating at the end gives back exactly the median of the per-round ratios - the transform costs nothing in interpretation.

2. Rounds where either side measured zero are dropped. The logarithm of zero is not a large number, it is negative infinity, and a single one of those makes every statistic downstream meaningless. A round with no measurable time carries no ratio, so it carries no information.

Estimating and reporting

1. The median as the point estimate. Its breakdown point is 50%: half the rounds can be arbitrarily corrupted before it moves at all. A mean has a breakdown point of zero - one descheduled round is enough to shift it - and benchmark timings are exactly the kind of data that produces the occasional wild value. This is the same reasoning behind nanobench reporting a median and an err% rather than a mean and a standard deviation.

2. The sign test for the interval. The interval is a pair of order statistics: with n rounds, the k-th smallest and k-th largest log ratios, where k is the largest one whose binomial tail still fits in 2.5%. It assumes the rounds are independent and nothing else - no distribution shape, no symmetry, no finite variance, no asymptotics - and it is exact at every n rather than approximately right for large ones. It is also deterministic: there is no resampling anywhere, so the same measurements always give the same interval.

3. Significance is the interval excluding 1. isSignificant() asks only that, which is the same thing as a two-sided test at 5% - and it is reported as an interval rather than a p-value because the interval says how big the difference is as well as whether there is one.

4. Tied rounds are counted and reported. When both sides land on the same tick, that round says the clock could not tell them apart, which is different information from the two being equally fast. Any median-based interval collapses to zero width when most rounds tie, so the count is what distinguishes a real 1.00x .. 1.00x from a measurement that never had the resolution.

What was rejected

The interval was the hard choice. Measured on right-skewed differences whose true median is exactly zero - which is what paired timings look like when one side has the heavier tail, since an operation can be arbitrarily slower but never faster than its floor:

Method	Coverage	Why
sign test	96.7%	Used. Assumes independence and nothing else. Slightly conservative, and about 10% wider than the bootstrap - which is the right direction to err for a number that ends up in a pull request.
percentile bootstrap	95.0%	Wants its own asymptotics, converges slowly for a median in particular, and collapses to zero width once a majority of rounds tie. Needs a seed, so the reported number depends on it.
Wilcoxon / Hodges-Lehmann	91.5%	Narrowest, and wrong here: it wants the differences symmetric about their median, which is precisely what skewed timings do not give. It also estimates the pseudomedian, so its interval would not be an interval for the number being reported.
t-interval	n/a	Wants normality and a finite variance, and has a breakdown point of zero.

Serial correlation, and why warmup is not the answer

The one assumption every method above shares is that the rounds are independent. Benchmark rounds have every reason not to be: frequency and thermal state persist across them, so a slow round makes the next one more likely to be slow too. Positive autocorrelation would make any of these intervals narrower than they should be.

Measured over 200 rounds of two identical operations, lag-1 autocorrelation:

Series	lag-1 correlation
raw per-round times	+0.10
paired log ratios	-0.08 .. -0.01
paired, first 20 rounds discarded	-0.07 .. -0.01

The correlation is real, and it is in the raw times. It is not in the paired differences, because that is what pairing is for: the drift is common to both sides of a round and subtracts out. What is left is slightly negative, which makes the interval conservative rather than too narrow - consistent with the false-positive rate measured below the nominal 5% rather than above it.

Note

This is also why warmup() is not the fix it looks like. Discarding the first twenty rounds - a warmup by another name - moves none of the numbers above, because calibration has already run each side for about a full epoch before the first round starts. compare() does honor warmup() if you set it, but do not expect it to buy an honest interval that pairing has not already bought.

The measurements above are one machine and one workload. A laptop that thermally throttles under sustained load could look different, and a block bootstrap would be the principled answer if it ever does.

Warning

Interleaving is a different measurement from running either side alone. Each alternative runs with the other’s cache and branch predictor state. That is usually the more honest number for “which should I ship”, and it is the wrong number for “how fast is this in isolation” - use run() for that.

Note

This resolves differences down to about 0.1%, which means it also resolves differences caused by where the compiler happened to put the code. Two distinct functions doing identical arithmetic report a difference about 10% of the time. That is a real difference - just not the one you meant to measure - so treat a sub-percent result as a question about code layout rather than about the algorithm.

Note

The interval assumes the rounds are independent, and strictly they are not: thermal and frequency state persist across them. Interleaving removes drift from each paired difference but does not make the differences independent, and positive autocorrelation makes any such interval narrower than it should be. In practice the measured error rate lands slightly below the nominal 5% rather than above it, but the assumption is worth knowing about before trusting a very tight interval.

Asymptotic Complexity

It is possible to calculate asymptotic complexity (Big O) from multiple runs of a benchmark. Run the benchmark with different complexity N, then nanobench can calculate the best fitting curve.

The following example finds out the asymptotic complexity of std::set’s find().

tutorial_complexity_set.cpp

#include <nanobench.h>
#include <thirdparty/doctest/doctest.h>

#include <iostream>
#include <set>

// NOLINTNEXTLINE
TEST_CASE("tutorial_complexity_set_find") {
    // Create a single benchmark instance that is used in multiple benchmark
    // runs, with different settings for complexityN.
    ankerl::nanobench::Bench bench;

    // a RNG to generate input data
    ankerl::nanobench::Rng rng;

    std::set<uint64_t> set;

    // Running the benchmark multiple times, with different number of elements
    for (auto setSize :
         {10U, 20U, 50U, 100U, 200U, 500U, 1000U, 2000U, 5000U, 10000U}) {

        // fill up the set with random data
        while (set.size() < setSize) {
            set.insert(rng());
        }

        // Run the benchmark, provide setSize as the scaling variable.
        bench.complexityN(set.size()).run("std::set find", [&] {
            ankerl::nanobench::doNotOptimizeAway(set.find(rng()));
        });
    }

    // calculate BigO complexy best fit and print the results
    std::cout << bench.complexityBigO() << std::endl;
}

The loop runs the benchmark 10 times, with different set sizes from 10 to 10k.

Note

Each of the 10 benchmark runs automatically scales the number of iterations so results are still fast and accurate. In total the whole test takes about 90ms.

The Bench object holds the benchmark results of the 10 benchmark runs. Each benchmark is recorded with a different setting for complexityN.

After the benchmark prints the benchmark results, we calculate & print the Big O of the most important complexity functions. std::cout << bench.complexityBigO() << std::endl; prints e.g. this markdown table:

|   coefficient |   err% | complexity
|--------------:|-------:|------------
|   6.66562e-09 |  29.1% | O(log n)
|   1.47588e-11 |  58.3% | O(n)
|   1.10742e-12 |  62.6% | O(n log n)
|   5.15683e-08 |  63.8% | O(1)
|   1.40387e-15 |  78.7% | O(n^2)
|   1.32792e-19 |  85.7% | O(n^3)

The table is sorted, best fitting complexity function first. So \(\mathcal{O}(\log{}n)\) provides the best approximation for the complexity. Interestingly, in that case error compared to \(\mathcal{O}(n)\) is not very large, which can be an indication that even though the red-black tree should theoretically have logarithmic complexity, in practices that is not perfectly the case.

Rendering Mustache-like Templates

Nanobench comes with a powerful Mustache-like template mechanism to process the benchmark results into all kinds of formats. You can find a full description of all possible tags at ankerl::nanobench::render().

Several preconfigured format exist in the namespace ankerl::nanobench::templates. Rendering these templates can be done with either ankerl::nanobench::render(), or directly with ankerl::nanobench::Bench::render().

The following example shows how to use the CSV - Comma-Separated Values template, without writing the standard output.

tutorial_render_simple.cpp

#include <nanobench.h>
#include <thirdparty/doctest/doctest.h>

#include <atomic>
#include <iostream>

// NOLINTNEXTLINE
TEST_CASE("tutorial_render_simple") {
    std::atomic<int> x(0);

    ankerl::nanobench::Bench()
        .output(nullptr)
        .run("std::vector",
             [&] {
                 ++x;
             })
        .render(ankerl::nanobench::templates::csv(), std::cout);
}

In line 11 we call Bench::output() with nullptr, thus disabling the standard output.

After the benchmark we directly call Bench::render() in line 16. Here we use the CSV template, and write the rendered output to std::cout. When running, we get just the CSV output to the console which looks like this:

"title";"name";"unit";"batch";"elapsed";"error %";"instructions";"branches";"branch misses";"total"
"benchmark";"std::vector";"op";1;6.51982200647249e-09;8.26465858909014e-05;23.0034662045061;5;0.00116867939228672;0.000171959

Nanobench comes with a few preconfigured templates, residing in the namespace ankerl::nanobench::templates. To demonstrate what these templates can do, here is a simple example that benchmarks two random generators std::mt19937_64 and std::knuth_b and prints both the template and the rendered output:

#include <nanobench.h>
#include <thirdparty/doctest/doctest.h>

#include <fstream>
#include <random>

namespace {

void gen(std::string const& typeName, char const* mustacheTemplate,
         ankerl::nanobench::Bench const& bench) {

    std::ofstream templateOut("mustache.template." + typeName);
    templateOut << mustacheTemplate;

    std::ofstream renderOut("mustache.render." + typeName);
    ankerl::nanobench::render(mustacheTemplate, bench, renderOut);
}

} // namespace

// NOLINTNEXTLINE
TEST_CASE("tutorial_mustache") {
    ankerl::nanobench::Bench bench;
    bench.title("Benchmarking std::mt19937_64 and std::knuth_b");

    // NOLINTNEXTLINE(cert-msc32-c,cert-msc51-cpp)
    std::mt19937_64 rng1;
    bench.run("std::mt19937_64", [&] {
        ankerl::nanobench::doNotOptimizeAway(rng1());
    });

    // NOLINTNEXTLINE(cert-msc32-c,cert-msc51-cpp)
    std::knuth_b rng2;
    bench.run("std::knuth_b", [&] {
        ankerl::nanobench::doNotOptimizeAway(rng2());
    });

    gen("json", ankerl::nanobench::templates::json(), bench);
    gen("html", ankerl::nanobench::templates::htmlBoxplot(), bench);
    gen("csv", ankerl::nanobench::templates::csv(), bench);
}

Nanobench allows to specify further context information, which may be accessed using {{context(name)}} where name names a variable defined via Bench::context().

#include <nanobench.h>
#include <thirdparty/doctest/doctest.h>

#include <cmath>
#include <iostream>

namespace {

template <typename T>
void fma_bench() {
    T x(1);
    T y(2);
    T z(3);
    z = std::fma(x, y, z);
    ankerl::nanobench::doNotOptimizeAway(z);
}

template <typename T>
void plus_eq() {
    T x(1);
    T y(2);
    T z(3);
    z += x * y;
    ankerl::nanobench::doNotOptimizeAway(z);
}

char const* csv() {
    return R"DELIM("title";"name";"scalar";"foo";"elapsed";"total"
{{#result}}"{{title}}";"{{name}}";"{{context(scalar)}}";"{{context(foo)}}";{{median(elapsed)}};{{sumProduct(iterations, elapsed)}}
{{/result}})DELIM";
}

} // namespace

// NOLINTNEXTLINE
TEST_CASE("tutorial_context") {
    ankerl::nanobench::Bench bench;
    bench.title("Addition").output(nullptr);
    bench.context("scalar", "f32")
        .context("foo", "bar")
        .run("+=", plus_eq<float>)
        .run("fma", fma_bench<float>);
    bench.context("scalar", "f64")
        .context("foo", "baz")
        .run("+=", plus_eq<double>)
        .run("fma", fma_bench<double>);
    bench.render(csv(), std::cout);
    // Changing the title resets the results, but not the context:
    bench.title("New Title");
    bench.run("+=", plus_eq<float>);
    bench.render(csv(), std::cout);
    CHECK_EQ(bench.results().front().context("foo"), "baz"); // != bar
    // The context has to be reset manually, which causes render to fail:
    bench.title("Yet Another Title").clearContext();
    bench.run("+=", plus_eq<float>);

    // NOLINTNEXTLINE(llvm-else-after-return,readability-else-after-return)
    CHECK_THROWS(bench.render(csv(), std::cout));
}

Time Units in Templates

elapsed is in seconds, which is rarely what a report should contain, and the template language has no arithmetic to rescale it. So the time measure also comes in elapsedms, elapsedus and elapsedns:

"name";"min_ms";"median_ns"
{{#result}}"{{name}}";{{minimum(elapsedms)}};{{median(elapsedns)}}
{{/result}}

All of them are per iteration, exactly like elapsed. {{medianAbsolutePercentError(...)}} is a relative error, so it is the same number whichever you ask for.

Note

Two things that are easy to mix up here:

• {{unit}} renders Bench::unit(), which is what a batch counts - op by default, or byte, MFlop and so on. It is not a time unit, which is why it prints op next to a value in seconds.

• Bench::timeUnit() only changes the ns/op column of the console table. It does not affect what templates render; use the suffixed measures above for that.

CSV - Comma-Separated Values

The function ankerl::nanobench::templates::csv() provides this template:

"title";"name";"unit";"batch";"elapsed";"error %";"instructions";"branches";"branch misses";"total"
{{#result}}"{{title}}";"{{name}}";"{{unit}}";{{batch}};{{median(elapsed)}};{{medianAbsolutePercentError(elapsed)}};{{median(instructions)}};{{median(branchinstructions)}};{{median(branchmisses)}};{{sumProduct(iterations, elapsed)}}
{{/result}}

This generates a compact CSV file, where entries are separated by a semicolon ;. Run with the example, I get this output:

"title";"name";"unit";"batch";"elapsed";"error %";"instructions";"branches";"branch misses";"total"
"Benchmarking std::mt19937_64 and std::knuth_b";"std::mt19937_64";"op";1;2.54441805225653e-08;0.0236579384033733;125.989678899083;16.7645714285714;0.564133016627078;0.000218811
"Benchmarking std::mt19937_64 and std::knuth_b";"std::knuth_b";"op";1;3.19013867488444e-08;0.00091350764819687;170.013008130081;28;0.0031104199066874;0.000217248

Rendered as CSV table:

title	name	unit	batch	elapsed	error %	instructions	branches	branch misses	total
Benchmarking std::mt19937_64 and std::knuth_b	std::mt19937_64	op	1	2.54441805225653e-08	0.0236579384033733	125.989678899083	16.7645714285714	0.564133016627078	0.000218811
Benchmarking std::mt19937_64 and std::knuth_b	std::knuth_b	op	1	3.19013867488444e-08	0.00091350764819687	170.013008130081	28	0.0031104199066874	0.000217248

Note that the CSV template doesn’t provide all the data that is available.

HTML Box Plots

With the template ankerl::nanobench::templates::htmlBoxplot() you get a plotly based HTML output which generates a boxplot of the runtime. The template is rather simple.

<html>

<head>
    <script src="https://cdn.plot.ly/plotly-latest.min.js"></script>
</head>

<body>
    <div id="myDiv"></div>
    <script>
        var data = [
            {{#result}}{
                name: '{{name}}',
                y: [{{#measurement}}{{elapsed}}{{^-last}}, {{/last}}{{/measurement}}],
            },
            {{/result}}
        ];
        var title = '{{title}}';

        data = data.map(a => Object.assign(a, { boxpoints: 'all', pointpos: 0, type: 'box' }));
        var layout = { title: { text: title }, showlegend: false, yaxis: { title: 'time per unit', rangemode: 'tozero', autorange: true } }; Plotly.newPlot('myDiv', data, layout, {responsive: true});
    </script>
</body>

</html>

This generates a nice interactive boxplot, which gives a nice visual showcase of the runtime performance of the evaluated benchmarks. Each epoch is visualized as a dot, and the boxplot itself shows median, percentiles, and outliers. You’ll might want to increase the default number of epochs for an even better visualization result.

JSON - JavaScript Object Notation

The ankerl::nanobench::templates::json() template gives everything, all data that is available, from all runs. The template is therefore quite complex:

{
    "results": [
{{#result}}        {
            "title": "{{title}}",
            "name": "{{name}}",
            "unit": "{{unit}}",
            "batch": {{batch}},
            "complexityN": {{complexityN}},
            "epochs": {{epochs}},
            "clockResolution": {{clockResolution}},
            "clockResolutionMultiple": {{clockResolutionMultiple}},
            "maxEpochTime": {{maxEpochTime}},
            "minEpochTime": {{minEpochTime}},
            "minEpochIterations": {{minEpochIterations}},
            "epochIterations": {{epochIterations}},
            "warmup": {{warmup}},
            "relative": {{relative}},
            "median(elapsed)": {{median(elapsed)}},
            "medianAbsolutePercentError(elapsed)": {{medianAbsolutePercentError(elapsed)}},
            "median(instructions)": {{median(instructions)}},
            "medianAbsolutePercentError(instructions)": {{medianAbsolutePercentError(instructions)}},
            "median(cpucycles)": {{median(cpucycles)}},
            "median(contextswitches)": {{median(contextswitches)}},
            "median(pagefaults)": {{median(pagefaults)}},
            "median(branchinstructions)": {{median(branchinstructions)}},
            "median(branchmisses)": {{median(branchmisses)}},
            "totalTime": {{sumProduct(iterations, elapsed)}},
            "measurements": [
{{#measurement}}                {
                    "iterations": {{iterations}},
                    "elapsed": {{elapsed}},
                    "pagefaults": {{pagefaults}},
                    "cpucycles": {{cpucycles}},
                    "contextswitches": {{contextswitches}},
                    "instructions": {{instructions}},
                    "branchinstructions": {{branchinstructions}},
                    "branchmisses": {{branchmisses}}
                }{{^-last}},{{/-last}}
{{/measurement}}            ]
        }{{^-last}},{{/-last}}
{{/result}}    ]
}

This also gives the data from each separate ankerl::nanobench::Bench::epochs(), not just the accumulated data as in the CSV template.

{
    "results": [
        {
            "title": "Benchmarking std::mt19937_64 and std::knuth_b",
            "name": "std::mt19937_64",
            "unit": "op",
            "batch": 1,
            "complexityN": -1,
            "epochs": 11,
            "clockResolution": 1.8e-08,
            "clockResolutionMultiple": 1000,
            "maxEpochTime": 0.1,
            "minEpochTime": 0,
            "minEpochIterations": 1,
            "warmup": 0,
            "relative": 0,
            "median(elapsed)": 2.54441805225653e-08,
            "medianAbsolutePercentError(elapsed)": 0.0236579384033733,
            "median(instructions)": 125.989678899083,
            "medianAbsolutePercentError(instructions)": 0.035125448044942,
            "median(cpucycles)": 81.3479809976247,
            "median(contextswitches)": 0,
            "median(pagefaults)": 0,
            "median(branchinstructions)": 16.7645714285714,
            "median(branchmisses)": 0.564133016627078,
            "totalTime": 0.000218811,
            "measurements": [
                {
                    "iterations": 875,
                    "elapsed": 2.54708571428571e-08,
                    "pagefaults": 0,
                    "cpucycles": 81.472,
                    "contextswitches": 0,
                    "instructions": 125.885714285714,
                    "branchinstructions": 16.7645714285714,
                    "branchmisses": 0.574857142857143
                },
                {
                    "iterations": 809,
                    "elapsed": 2.58467243510507e-08,
                    "pagefaults": 0,
                    "cpucycles": 82.5290482076638,
                    "contextswitches": 0,
                    "instructions": 128.771322620519,
                    "branchinstructions": 17.0296662546354,
                    "branchmisses": 0.582200247218789
                },
                {
                    "iterations": 737,
                    "elapsed": 2.24097693351425e-08,
                    "pagefaults": 0,
                    "cpucycles": 71.6431478968792,
                    "contextswitches": 0,
                    "instructions": 118.374491180461,
                    "branchinstructions": 15.9470827679783,
                    "branchmisses": 0.417910447761194
                },
                {
                    "iterations": 872,
                    "elapsed": 2.53405963302752e-08,
                    "pagefaults": 0,
                    "cpucycles": 80.9896788990826,
                    "contextswitches": 0,
                    "instructions": 125.989678899083,
                    "branchinstructions": 16.7580275229358,
                    "branchmisses": 0.563073394495413
                },
                {
                    "iterations": 834,
                    "elapsed": 2.59256594724221e-08,
                    "pagefaults": 0,
                    "cpucycles": 82.7661870503597,
                    "contextswitches": 0,
                    "instructions": 127.635491606715,
                    "branchinstructions": 16.9352517985612,
                    "branchmisses": 0.575539568345324
                },
                {
                    "iterations": 772,
                    "elapsed": 2.25310880829016e-08,
                    "pagefaults": 0,
                    "cpucycles": 72.0129533678757,
                    "contextswitches": 0,
                    "instructions": 117.108808290155,
                    "branchinstructions": 15.8341968911917,
                    "branchmisses": 0.405440414507772
                },
                {
                    "iterations": 842,
                    "elapsed": 2.54441805225653e-08,
                    "pagefaults": 0,
                    "cpucycles": 81.3479809976247,
                    "contextswitches": 0,
                    "instructions": 127.266033254157,
                    "branchinstructions": 16.8859857482185,
                    "branchmisses": 0.564133016627078
                },
                {
                    "iterations": 792,
                    "elapsed": 2.20126262626263e-08,
                    "pagefaults": 0,
                    "cpucycles": 70.3623737373737,
                    "contextswitches": 0,
                    "instructions": 116.420454545455,
                    "branchinstructions": 15.7588383838384,
                    "branchmisses": 0.396464646464646
                },
                {
                    "iterations": 757,
                    "elapsed": 2.63870541611625e-08,
                    "pagefaults": 0,
                    "cpucycles": 84.332892998679,
                    "contextswitches": 0,
                    "instructions": 131.462351387054,
                    "branchinstructions": 17.334214002642,
                    "branchmisses": 0.618229854689564
                },
                {
                    "iterations": 850,
                    "elapsed": 2.23305882352941e-08,
                    "pagefaults": 0,
                    "cpucycles": 71.3505882352941,
                    "contextswitches": 0,
                    "instructions": 114.629411764706,
                    "branchinstructions": 15.5823529411765,
                    "branchmisses": 0.392941176470588
                },
                {
                    "iterations": 774,
                    "elapsed": 2.60607235142119e-08,
                    "pagefaults": 0,
                    "cpucycles": 83.1679586563308,
                    "contextswitches": 0,
                    "instructions": 130.576227390181,
                    "branchinstructions": 17.2635658914729,
                    "branchmisses": 0.590439276485788
                }
            ]
        },
        {
            "title": "Benchmarking std::mt19937_64 and std::knuth_b",
            "name": "std::knuth_b",
            "unit": "op",
            "batch": 1,
            "complexityN": -1,
            "epochs": 11,
            "clockResolution": 1.8e-08,
            "clockResolutionMultiple": 1000,
            "maxEpochTime": 0.1,
            "minEpochTime": 0,
            "minEpochIterations": 1,
            "warmup": 0,
            "relative": 0,
            "median(elapsed)": 3.19013867488444e-08,
            "medianAbsolutePercentError(elapsed)": 0.00091350764819687,
            "median(instructions)": 170.013008130081,
            "medianAbsolutePercentError(instructions)": 4.11992392254248e-06,
            "median(cpucycles)": 101.973254086181,
            "median(contextswitches)": 0,
            "median(pagefaults)": 0,
            "median(branchinstructions)": 28,
            "median(branchmisses)": 0.0031104199066874,
            "totalTime": 0.000217248,
            "measurements": [
                {
                    "iterations": 568,
                    "elapsed": 3.2137323943662e-08,
                    "pagefaults": 0,
                    "cpucycles": 102.55985915493,
                    "contextswitches": 0,
                    "instructions": 170.014084507042,
                    "branchinstructions": 28,
                    "branchmisses": 0.00528169014084507
                },
                {
                    "iterations": 576,
                    "elapsed": 3.19305555555556e-08,
                    "pagefaults": 0,
                    "cpucycles": 102.059027777778,
                    "contextswitches": 0,
                    "instructions": 170.013888888889,
                    "branchinstructions": 28,
                    "branchmisses": 0.00347222222222222
                },
                {
                    "iterations": 643,
                    "elapsed": 3.18973561430793e-08,
                    "pagefaults": 0,
                    "cpucycles": 101.973561430793,
                    "contextswitches": 0,
                    "instructions": 170.012441679627,
                    "branchinstructions": 28,
                    "branchmisses": 0.0031104199066874
                },
                {
                    "iterations": 591,
                    "elapsed": 3.1912013536379e-08,
                    "pagefaults": 0,
                    "cpucycles": 101.944162436548,
                    "contextswitches": 0,
                    "instructions": 170.013536379019,
                    "branchinstructions": 28,
                    "branchmisses": 0.00169204737732657
                },
                {
                    "iterations": 673,
                    "elapsed": 3.19049034175334e-08,
                    "pagefaults": 0,
                    "cpucycles": 101.973254086181,
                    "contextswitches": 0,
                    "instructions": 170.011887072808,
                    "branchinstructions": 28,
                    "branchmisses": 0.00297176820208024
                },
                {
                    "iterations": 649,
                    "elapsed": 3.19013867488444e-08,
                    "pagefaults": 0,
                    "cpucycles": 101.850539291217,
                    "contextswitches": 0,
                    "instructions": 170.012326656394,
                    "branchinstructions": 28,
                    "branchmisses": 0.00308166409861325
                },
                {
                    "iterations": 606,
                    "elapsed": 3.18547854785479e-08,
                    "pagefaults": 0,
                    "cpucycles": 101.83498349835,
                    "contextswitches": 0,
                    "instructions": 170.013201320132,
                    "branchinstructions": 28,
                    "branchmisses": 0.0033003300330033
                },
                {
                    "iterations": 650,
                    "elapsed": 3.18769230769231e-08,
                    "pagefaults": 0,
                    "cpucycles": 101.898461538462,
                    "contextswitches": 0,
                    "instructions": 170.012307692308,
                    "branchinstructions": 28,
                    "branchmisses": 0.00307692307692308
                },
                {
                    "iterations": 615,
                    "elapsed": 3.18520325203252e-08,
                    "pagefaults": 0,
                    "cpucycles": 101.858536585366,
                    "contextswitches": 0,
                    "instructions": 170.013008130081,
                    "branchinstructions": 28,
                    "branchmisses": 0.0032520325203252
                },
                {
                    "iterations": 579,
                    "elapsed": 3.18618307426598e-08,
                    "pagefaults": 0,
                    "cpucycles": 101.989637305699,
                    "contextswitches": 0,
                    "instructions": 170.013816925734,
                    "branchinstructions": 28,
                    "branchmisses": 0.00345423143350604
                },
                {
                    "iterations": 657,
                    "elapsed": 3.19558599695586e-08,
                    "pagefaults": 0,
                    "cpucycles": 102.229832572298,
                    "contextswitches": 0,
                    "instructions": 170.012176560122,
                    "branchinstructions": 28,
                    "branchmisses": 0.0030441400304414
                }
            ]
        }
    ]
}

pyperf - Python pyperf module Output

Pyperf is a powerful tool for benchmarking and system tuning, and it can also analyze benchmark results. This template allows generation of output so it can be used for further analysis with pyperf.

Note

Pyperf supports only a single benchmark result per generated output, so it is best to create a new Bench object for each benchmark.

The template looks like this. Note that it directly makes use of {{#measurement}}, which is only possible when there is a single result in the benchmark.

{
    "benchmarks": [
        {
            "runs": [
                {
                    "values": [
{{#measurement}}                        {{elapsed}}{{^-last}},
{{/last}}{{/measurement}}
                    ]
                }
            ]
        }
    ],
    "metadata": {
        "loops": {{sum(iterations)}},
        "inner_loops": {{batch}},
        "name": "{{title}}",
        "unit": "second"
    },
    "version": "1.0"
}

Here is an example that generates pyperf compatible output for a benchmark that shuffles a vector:

example_pyperf.cpp

#include <nanobench.h>
#include <thirdparty/doctest/doctest.h>

#include <algorithm>
#include <fstream>
#include <random>

// NOLINTNEXTLINE
TEST_CASE("shuffle_pyperf") {
    std::vector<uint64_t> data(500, 0); // input data for shuffling

    // NOLINTNEXTLINE(cert-msc32-c,cert-msc51-cpp)
    std::default_random_engine defaultRng(123);
    std::ofstream fout1("pyperf_shuffle_std.json");
    ankerl::nanobench::Bench()
        .epochs(100)
        .run("std::shuffle with std::default_random_engine",
             [&]() {
                 std::shuffle(data.begin(), data.end(), defaultRng);
             })
        .render(ankerl::nanobench::templates::pyperf(), fout1);

    std::ofstream fout2("pyperf_shuffle_nanobench.json");
    ankerl::nanobench::Rng rng(123);
    ankerl::nanobench::Bench()
        .epochs(100)
        .run("ankerl::nanobench::Rng::shuffle",
             [&]() {
                 rng.shuffle(data);
             })
        .render(ankerl::nanobench::templates::pyperf(), fout2);
}

This benchmark run creates the two files pyperf_shuffle_std.json and pyperf_shuffle_nanobench.json. Here are some of the analysis you can do:

Show Benchmark Statistics

Output from python3 -m pyperf stats pyperf_shuffle_std.json:

Total duration: 364 ms
Raw value minimum: 3.57 ms
Raw value maximum: 4.21 ms

Number of calibration run: 0
Number of run with values: 1
Total number of run: 1

Number of warmup per run: 0
Number of value per run: 100
Loop iterations per value: 100
Total number of values: 100

Minimum:         35.7 us
Median +- MAD:   36.2 us +- 0.2 us
Mean +- std dev: 36.4 us +- 0.9 us
Maximum:         42.1 us

  0th percentile: 35.7 us (-2% of the mean) -- minimum
  5th percentile: 35.8 us (-2% of the mean)
 25th percentile: 36.1 us (-1% of the mean) -- Q1
 50th percentile: 36.2 us (-0% of the mean) -- median
 75th percentile: 36.4 us (+0% of the mean) -- Q3
 95th percentile: 36.7 us (+1% of the mean)
100th percentile: 42.1 us (+16% of the mean) -- maximum

Number of outlier (out of 35.6 us..36.9 us): 4

Show a Histogram

It’s often interesting to see a histogram, especially to visually find out if there are outliers involved. Run python3 -m pyperf hist pyperf_shuffle_std.json produces this output

35.7 us: 21 ######################################
36.0 us: 33 ############################################################
36.3 us: 37 ###################################################################
36.6 us:  5 #########
36.9 us:  0 |
37.2 us:  1 ##
37.5 us:  0 |
37.8 us:  0 |
38.1 us:  0 |
38.4 us:  0 |
38.7 us:  0 |
39.0 us:  0 |
39.3 us:  0 |
39.6 us:  1 ##
39.9 us:  0 |
40.2 us:  0 |
40.5 us:  1 ##
40.8 us:  0 |
41.1 us:  0 |
41.5 us:  0 |
41.8 us:  0 |
42.1 us:  1 ##

Compare Results

We have generated two results in the above examples, and we can compare them easily with python3 -m pyperf compare_to a.json b.json:

+-----------+--------------------+------------------------------+
| Benchmark | pyperf_shuffle_std | pyperf_shuffle_nanobench     |
+===========+====================+==============================+
| benchmark | 36.4 us            | 11.2 us: 3.24x faster (-69%) |
+-----------+--------------------+------------------------------+

For more information of pyperfs analysis capability, please see pyperf - Analyze benchmark results.