Benchmark Infrastructure

[arthur]

Similar to the Test infrastructure thread, I’d like to field recommendations for benchmarking libraries and requirements we hold over them.

To the latter point, if I understand correctly, portability and easy vendoring of the beman library itself is a primary concern. Relevant “beman standards”:

• cmake.use_find_package
• came.passive_projects
• make.passive_targets

I wonder if we might want a benchmark analogue of

• cmake.skip_tests
• directory.tests

? Or skipping testing is sufficient to mark the benchmarks should be skipped too? (I have seen them separately specified in the wild, and certainly was thankful when it was so.)

Given these constraints, I don’t know that this renders certain options inoperable (I haven’t gone through these in detail), but at least these are some benchmarking libraries that seem practical:

• martinus/nanobench
• catch2 built-in benchmarks
• sheredom/ubench.h
• google/benchmark – well trodden, a bit heavy…

What guidance or recommendations do you think we should offer on the topic of benchmarking?

[ednolan]

To my understanding the only Beman library that has implemented benchmarks so far is Patrick’s beman.any_view:

tests/beman/any_view/CMakeLists.txt

It uses Google Benchmark.

So essentially the only existing guidance we can currently offer is “do what any_view does,” but this is definitely still an area where we welcome experimentation and proposals.