Test Set
I’ve implemented the three different benchmarks: Fast, Slow, Fluctuating in several frameworks for comparison.
- Fast
Benchmarks
x += x, starting from 1. This is a single instruction, and prone to be optimized away.- Slow
Benchmarks
std::this_thread::sleep_for(10ms). For a microbenchmark this is very slow, and it is interesting how the framework’s autotuning deals with this.- Fluctuating
A microbenchmark where each evaluation takes a different time. This randomly fluctuating runtime is achieved by randomly producing 0-255 random numbers with
std::mt19937_64.
All benchmarks are run on an i7-8700 CPU locked at 3.2GHz, using pyperf system tune.
Runtime
I wrote a little timing tool that measures how long exactly it takes to print benchmark output to the screen. With this I have measured the runtimes of major benchmarking frameworks which support automatic tuning of the number of iterations: Google Benchmark, Catch2, nonius, sltbench, and of course nanobench.
Benchmarking Framework |
Fast |
Slow |
Fluctuating |
Overhead |
total |
|---|---|---|---|---|---|
Google Benchmark |
0.367 |
11.259 |
0.825 |
0.000 |
12.451 |
Catch2 |
1.004 |
2.074 |
0.966 |
1.737 |
5.782 |
nonius |
0.741 |
1.815 |
0.740 |
1.715 |
5.010 |
sltbench |
0.202 |
0.204 |
0.203 |
3.001 |
3.610 |
nanobench |
0.079 |
0.112 |
0.000 |
0.001 |
0.192 |
Nanobench is clearly the fastest autotuning benchmarking framework, by an enormous margin.
Implementations & Output
nanobench
Sourcecode
1// https://github.com/martinus/nanobench
2// g++ -O2 -I../../include main.cpp -o m
3
4#define ANKERL_NANOBENCH_IMPLEMENT
5#include <nanobench.h>
6
7#include <chrono>
8#include <random>
9#include <thread>
10
11int main(int, char**) {
12 uint64_t x = 1;
13 ankerl::nanobench::Bench().run("x += x", [&]() {
14 ankerl::nanobench::doNotOptimizeAway(x += x);
15 });
16
17 ankerl::nanobench::Bench().run("sleep 10ms", [&]() {
18 std::this_thread::sleep_for(std::chrono::milliseconds(10));
19 });
20
21 std::random_device dev;
22 std::mt19937_64 rng(dev());
23 ankerl::nanobench::Bench().run("random fluctuations", [&]() {
24 // each run, perform a random number of rng calls
25 auto iterations = rng() & UINT64_C(0xff);
26 for (uint64_t i = 0; i < iterations; ++i) {
27 (void)rng();
28 }
29 });
30}
Results
| ns/op | op/s | err% | ins/op | cyc/op | IPC | bra/op | miss% | total | benchmark
|--------------------:|--------------------:|--------:|----------------:|----------------:|-------:|---------------:|--------:|----------:|:----------
| 0.31 | 3,192,709,967.58 | 0.0% | 1.00 | 1.00 | 0.999 | 0.00 | 0.0% | 0.00 | `x += x`
| 10,149,086.00 | 98.53 | 0.1% | 45.00 | 2,394.00 | 0.019 | 9.00 | 88.9% | 0.11 | `sleep 10ms`
| 744.50 | 1,343,183.34 | 11.2% | 2,815.05 | 2,375.86 | 1.185 | 524.73 | 12.5% | 0.00 | :wavy_dash: `random fluctuations` (Unstable with ~23.3 iters. Increase `minEpochIterations` to e.g. 233)
Google Benchmark
Very feature rich, battle proven, but a bit aged. Requires google test. Get it here: Google Benchmark
Sourcecode
1#include "benchmark.h"
2
3#include <chrono>
4#include <random>
5#include <thread>
6
7// Build instructions: https://github.com/google/benchmark#installation
8// curl --output benchmark.h
9// https://raw.githubusercontent.com/google/benchmark/master/include/benchmark/benchmark.h
10// g++ -O2 main.cpp -Lgit/benchmark/build/src -lbenchmark -lpthread -o m
11void ComparisonFast(benchmark::State& state) {
12 uint64_t x = 1;
13 for (auto _ : state) {
14 x += x;
15 }
16 benchmark::DoNotOptimize(x);
17}
18BENCHMARK(ComparisonFast);
19
20void ComparisonSlow(benchmark::State& state) {
21 for (auto _ : state) {
22 std::this_thread::sleep_for(std::chrono::milliseconds(10));
23 }
24}
25BENCHMARK(ComparisonSlow);
26
27void ComparisonFluctuating(benchmark::State& state) {
28 std::random_device dev;
29 std::mt19937_64 rng(dev());
30 for (auto _ : state) {
31 // each run, perform a random number of rng calls
32 auto iterations = rng() & UINT64_C(0xff);
33 for (uint64_t i = 0; i < iterations; ++i) {
34 (void)rng();
35 }
36 }
37}
38BENCHMARK(ComparisonFluctuating);
39
40BENCHMARK_MAIN();
Results
Compiled & linked with
g++ -O2 main.cpp -L/home/martinus/git/benchmark/build/src -lbenchmark -lpthread -o gbench
executing it gives this result:
2019-10-12 12:03:25
Running ./gbench
Run on (12 X 4600 MHz CPU s)
CPU Caches:
L1 Data 32K (x6)
L1 Instruction 32K (x6)
L2 Unified 256K (x6)
L3 Unified 12288K (x1)
Load Average: 0.21, 0.55, 0.60
***WARNING*** CPU scaling is enabled, the benchmark real time measurements may be noisy and will incur extra overhead.
----------------------------------------------------------------
Benchmark Time CPU Iterations
----------------------------------------------------------------
ComparisonFast 0.313 ns 0.313 ns 1000000000
ComparisonSlow 10137913 ns 3920 ns 1000
ComparisonFluctuating 993 ns 992 ns 706946
Running the tests individually takes 0.365s, 11.274 sec, 0.828sec.
nonius
It gives lots of statistics, but seems a bit complicated to me. Not as straight forward as I’d like it. It shows lots of statistics, which makes the output a bit hard to read. I am not sure if it is still actively maintained. The homepage has been down for a while. Get it here: nonius
Sourcecode
1#define NONIUS_RUNNER
2#include <nonius/nonius_single.h++>
3
4// g++ -O2 main.cpp -pthread -I. -o m
5
6#include <chrono>
7#include <random>
8#include <thread>
9
10NONIUS_PARAM(X, UINT64_C(1))
11
12template <typename Fn>
13struct volatilize_fn {
14 Fn fn;
15 auto operator()() const -> decltype(fn()) {
16 volatile auto x = fn();
17 return x;
18 }
19};
20
21template <typename Fn>
22auto volatilize(Fn&& fn) -> volatilize_fn<typename std::decay<Fn>::type> {
23 return {std::forward<Fn>(fn)};
24}
25
26NONIUS_BENCHMARK("x += x", [](nonius::chronometer meter) {
27 auto x = meter.param<X>();
28 meter.measure(volatilize([&]() {
29 return x += x;
30 }));
31})
32
33NONIUS_BENCHMARK("sleep 10ms", [] {
34 std::this_thread::sleep_for(std::chrono::milliseconds(10));
35})
36
37NONIUS_BENCHMARK("random fluctuations", [](nonius::chronometer meter) {
38 std::random_device dev;
39 std::mt19937_64 rng(dev());
40 meter.measure([&] {
41 // each run, perform a random number of rng calls
42 auto iterations = rng() & UINT64_C(0xff);
43 for (uint64_t i = 0; i < iterations; ++i) {
44 (void)rng();
45 }
46 });
47})
Results
clock resolution: mean is 22.0426 ns (20480002 iterations)
new round for parameters
X = 1
benchmarking x += x
collecting 100 samples, 56376 iterations each, in estimated 0 ns
mean: 0.391109 ns, lb 0.391095 ns, ub 0.391135 ns, ci 0.95
std dev: 9.50619e-05 ns, lb 6.25215e-05 ns, ub 0.000167224 ns, ci 0.95
found 4 outliers among 100 samples (4%)
variance is unaffected by outliers
benchmarking sleep 10ms
collecting 100 samples, 1 iterations each, in estimated 1013.66 ms
mean: 10.1258 ms, lb 10.1189 ms, ub 10.1313 ms, ci 0.95
std dev: 31.1777 μs, lb 26.5814 μs, ub 35.4952 μs, ci 0.95
found 13 outliers among 100 samples (13%)
variance is unaffected by outliers
benchmarking random fluctuations
collecting 100 samples, 23 iterations each, in estimated 2.2724 ms
mean: 1016.26 ns, lb 991.161 ns, ub 1041.66 ns, ci 0.95
std dev: 128.963 ns, lb 109.803 ns, ub 159.509 ns, ci 0.95
found 2 outliers among 100 samples (2%)
variance is severely inflated by outliers
The tests individually take 0.713sec, 1.883sec, 0.819sec. Plus a startup overhead of 1.611sec.
Picobench
It took me a while to figure out that I have to configure the slow test, otherwise it would run for a looong time. The number of iterations is hardcoded, this library seems very basic. Get it here: picobench
Sourcecode
1#define PICOBENCH_IMPLEMENT_WITH_MAIN
2#include "picobench.hpp"
3
4#include <chrono>
5#include <initializer_list>
6#include <random>
7#include <thread>
8
9
10// https://github.com/iboB/picobench
11// g++ -O2 picobench.cpp -o pb
12
13PICOBENCH_SUITE("ComparisonFast");
14static void ComparisonFast(picobench::state& state) {
15 uint64_t x = 1;
16 for (auto _ : state) {
17 x += x;
18 }
19 state.set_result(x);
20}
21PICOBENCH(ComparisonFast);
22
23PICOBENCH_SUITE("ComparisonSlow");
24void ComparisonSlow(picobench::state& state) {
25 for (auto _ : state) {
26 std::this_thread::sleep_for(std::chrono::milliseconds(10));
27 }
28}
29PICOBENCH(ComparisonSlow).iterations({1, 2, 5, 10});
30
31PICOBENCH_SUITE("fluctuating");
32void ComparisonFluctuating(picobench::state& state) {
33 std::random_device dev;
34 std::mt19937_64 rng(dev());
35 for (auto _ : state) {
36 // each run, perform a random number of rng calls
37 auto iterations = rng() & UINT64_C(0xff);
38 for (uint64_t i = 0; i < iterations; ++i) {
39 (void)rng();
40 }
41 }
42}
43PICOBENCH(ComparisonFluctuating);
Results
ComparisonFast:
===============================================================================
Name (baseline is *) | Dim | Total ms | ns/op |Baseline| Ops/second
===============================================================================
ComparisonFast * | 8 | 0.000 | 6 | - |156862745.1
ComparisonFast * | 64 | 0.000 | 1 | - |512000000.0
ComparisonFast * | 512 | 0.000 | 0 | - |2560000000.0
ComparisonFast * | 4096 | 0.001 | 0 | - |3110098709.2
ComparisonFast * | 8192 | 0.003 | 0 | - |3141104294.5
===============================================================================
ComparisonSlow:
===============================================================================
Name (baseline is *) | Dim | Total ms | ns/op |Baseline| Ops/second
===============================================================================
ComparisonSlow * | 1 | 10.056 |10055959 | - | 99.4
ComparisonSlow * | 2 | 20.178 |10088773 | - | 99.1
ComparisonSlow * | 5 | 50.570 |10114054 | - | 98.9
ComparisonSlow * | 10 | 101.136 |10113643 | - | 98.9
===============================================================================
fluctuating:
===============================================================================
Name (baseline is *) | Dim | Total ms | ns/op |Baseline| Ops/second
===============================================================================
ComparisonFluctuating * | 8 | 0.012 | 1551 | - | 644485.6
ComparisonFluctuating * | 64 | 0.068 | 1057 | - | 945584.6
ComparisonFluctuating * | 512 | 0.565 | 1103 | - | 906222.0
ComparisonFluctuating * | 4096 | 4.469 | 1090 | - | 916619.4
ComparisonFluctuating * | 8192 | 9.003 | 1098 | - | 909957.2
===============================================================================
It doesn’t really make sense to provide runtime numbers here, because picobench just executes the given number of iterations, and that’s it. No autotuning.
Catch2
Catch2 is mostly a unit testing framework, and has recently integrated benchmarking faciliy. It is very easy to use, but does not seem too configurable. I find the way it writes the output very confusing. Get it here: Catch2
Sourcecode
1// https://github.com/catchorg/Catch2
2// g++ -O2 catch.cpp -o c
3
4#define CATCH_CONFIG_ENABLE_BENCHMARKING
5#define CATCH_CONFIG_MAIN
6#include "catch.hpp" // NOLINT
7
8#include <chrono>
9#include <random>
10#include <thread>
11
12TEST_CASE("comparison_fast"){
13 uint64_t x = 1;
14 BENCHMARK("x += x") {
15 return x += x;
16 };
17}
18
19TEST_CASE("comparison_slow") {
20 BENCHMARK("sleep 10ms") {
21 std::this_thread::sleep_for(std::chrono::milliseconds(10));
22 };
23}
24
25// NOLINTNEXTLINE(fuchsia-statically-constructed-objects,llvmlibc-implementation-in-namespace)
26TEST_CASE("comparison_fluctuating_v2") {
27 std::random_device dev;
28 std::mt19937_64 rng(dev());
29 BENCHMARK("random fluctuations") {
30 // each run, perform a random number of rng calls
31 auto iterations = rng() & UINT64_C(0xff);
32 for (uint64_t i = 0; i < iterations; ++i) {
33 (void)rng();
34 }
35 };
36}
Results
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
c is a Catch v2.9.2 host application.
Run with -? for options
-------------------------------------------------------------------------------
comparison_fast
-------------------------------------------------------------------------------
catch.cpp:12
...............................................................................
benchmark name samples iterations estimated
mean low mean high mean
std dev low std dev high std dev
-------------------------------------------------------------------------------
x += x 100 12414 1.2414 ms
1 ns 1 ns 1 ns
0 ns 0 ns 0 ns
-------------------------------------------------------------------------------
comparison_slow
-------------------------------------------------------------------------------
catch.cpp:19
...............................................................................
benchmark name samples iterations estimated
mean low mean high mean
std dev low std dev high std dev
-------------------------------------------------------------------------------
sleep 10ms 100 1 1.01319 s
10.1357 ms 10.1302 ms 10.1396 ms
23.539 us 18.061 us 29.575 us
-------------------------------------------------------------------------------
comparison_fluctuating_v2
-------------------------------------------------------------------------------
catch.cpp:25
...............................................................................
benchmark name samples iterations estimated
mean low mean high mean
std dev low std dev high std dev
-------------------------------------------------------------------------------
random fluctuations 100 28 2.3324 ms
827 ns 810 ns 844 ns
88 ns 79 ns 99 ns
===============================================================================
test cases: 3 | 3 passed
assertions: - none -
moodycamel::microbench
A very simple benchmarking tool, and an API that’s very similar to ankerl::nanobench. No autotuning,
no doNotOptimize, no output formatting. Get it here: moodycamel::microbench
Sourcecode
1#include "microbench.h"
2
3#include <chrono>
4#include <iostream>
5#include <random>
6#include <thread>
7
8// g++ -O2 -c systemtime.cpp
9// g++ -O2 -c microbench.cpp
10// g++ microbench.o systemtime.o -o mb
11int main(int, char**) {
12 // something fast
13 uint64_t x = 1;
14 std::cout << moodycamel::microbench(
15 [&]() {
16 x += x;
17 },
18 10000000, 51)
19 << " sec x += x (x==" << x << ")" << std::endl;
20
21 std::cout << moodycamel::microbench([&] {
22 std::this_thread::sleep_for(std::chrono::milliseconds(10));
23 }) << " sec sleep 10ms"
24 << std::endl;
25
26 std::random_device dev;
27 std::mt19937_64 rng(dev());
28 std::cout << moodycamel::microbench(
29 [&] {
30 // each run, perform a random number of rng calls
31 auto iterations = rng() & UINT64_C(0xff);
32 for (uint64_t i = 0; i < iterations; ++i) {
33 (void)rng();
34 }
35 },
36 1000, 51)
37 << " sec random fluctuations" << std::endl;
38}
Results
3.12506e-07 sec x += x (x==0)
10.056 sec sleep 10ms
0.000661384 sec random fluctuations
sltbench
C++ benchmark which seems to have similar intentions to nanonbech. It claims to be 4.7 times faster than googlebench.
It requires to be compiled and linked. I initially got a compile error because of missing <cstdint> include.
After that it compiled fine, and I created an example. I didn’t like that I had to use global variables for the state
that I needed in my ComparisonFast and ComparisonSlow benchmark. Get it
here: sltbench
Sourcecode
1#include <sltbench/Bench.h> // https://github.com/ivafanas/sltbench
2
3#include <chrono>
4#include <random>
5#include <thread>
6
7// cmake build as online instructions describes
8//
9// g++ -O3 -I/home/martinus/git/sltbench/install/include -c main.cpp
10// g++ -o m -L/home/martinus/git/sltbench/install/lib main.o -lsltbench
11
12uint64_t x = 1;
13void ComparisonFast() {
14 sltbench::DoNotOptimize(x += x);
15}
16
17SLTBENCH_FUNCTION(ComparisonFast);
18
19void ComparisonSlow() {
20 std::this_thread::sleep_for(std::chrono::milliseconds(10));
21}
22SLTBENCH_FUNCTION(ComparisonSlow);
23
24std::random_device dev;
25std::mt19937_64 rng(dev());
26
27void ComparisonFluctuating() {
28 // each run, perform a random number of rng calls
29 auto iterations = rng() & UINT64_C(0xff);
30 for (uint64_t i = 0; i < iterations; ++i) {
31 (void)rng();
32 }
33}
34SLTBENCH_FUNCTION(ComparisonFluctuating);
35
36SLTBENCH_MAIN();
Results
benchmark arg status time(ns)
ComparisonFast ok 1
ComparisonFluctuating ok 20
ComparisonSlow ok 10055943
Interestingly, the executable takes exactly 3 seconds startup time, then each benchmark runs for about 0.2 seconds.
Celero
Unfortunately I couldn’t get it working. I only got segmentation faults for my x += x benchmarks.
Get it here: celero
folly Benchmark
Facebook’s folly comes with benchmarking facility. It seems rather basic, but with good DoNotOptimizeAway
functionality. Honestly, I was too lazy to get this working. Too much installation hazzle. Get it here:
folly