Similar to the Test infrastructure thread, I’d like to field recommendations for benchmarking libraries and requirements we hold over them.
To the latter point, if I understand correctly, portability and easy vendoring of the beman library itself is a primary concern. Relevant “beman standards”:
? Or skipping testing is sufficient to mark the benchmarks should be skipped too? (I have seen them separately specified in the wild, and certainly was thankful when it was so.)
Given these constraints, I don’t know that this renders certain options inoperable (I haven’t gone through these in detail), but at least these are some benchmarking libraries that seem practical:
To my understanding the only Beman library that has implemented benchmarks so far is Patrick’s beman.any_view:
It uses Google Benchmark.
So essentially the only existing guidance we can currently offer is “do what any_view does,” but this is definitely still an area where we welcome experimentation and proposals.
I have a bunch of benchmarks, using Catch2, and some plain bulk timing, because I know I have to answer questions about specialized simd libs as well as existing facilities, and how much overhead the contemporary C++ adds.
It’s negative overhead, but it’s a fair question.
I have mixed feelings about benchmarks, though. Our reference libraries need to be correct and clear first. Avoiding designed in pessimization is important. But many high performance libraries also have really high semantic pre-conditions that are risky for the standard library. Especially as we’re shifting from performance to safety as the top criteria, away from our C heritage. But, I also know that standard library implementers are mostly not producing highly optimized implementations, they are working flat out producing good implementations of everything. They aren’t likely to do much better than what we write. Certainly not the magically good hand tuned asm I assumed existed before i saw all the code.
The operational problem is that good benchmarks are expensive. Cheap, micro, benchmarks most of the time don’t measure worthwhile conditions.
An area that I think I need to explore, though, is benchmarking cost of compilation.