Relevant: I just learned that microsoft/STL actually uses LLVM’s lit under the hood. A lead maintainer (also called STL) seems to like it.
From the STL README:
We rely on three test suites: std, tr1, and libcxx. We’ve partially ported std and tr1, and fully ported libcxx to run under lit using the various configurations/compilers we test internally.
Also, moderator of /r/cpp where I think we both learned this. Although I think Louis Dionne mentioned that all implementations use the llvm test suite - plus their own of course.
I think @dsankel wrote a paper about lit for beman, see this thread:
David’s conclusion was:
Conclusion
We’ve investigated making lit the recommended Beman testing framework. Unlike other frameworks it has support for compilation failure tests, but it has several drawbacks including lack of popularity, complexity, and ergonomics challenges. Our recommendation is to instead adopt a more popular framework, such as GTest, and supplement it with CMake functions that add compilation failure capabilities.
Many people don’t like gtest because it is not header-only. But relatively recently catch2 stopped being header only. Just thought I would mention it.
Now here’s my two pennysworth…
I am a fan of gtest. There, I’ve said it.
I understand that beman wants to minimise dependencies so maybe beman needs to try and standardize a unit test framework. For full power, mocking might be considered a part of that. It would be a big project, to be sure, but what a foundation it would provide for the rest of beman.
Hi @marlowa – welcome! I think the main reason we’re still discussing is that on thing that’s clear is that gtest, for a variety of reasons, won’t be used by standard library vendors. So for easy adoption we’d like something that would be at least easily convertible to the frameworks used by vendors. It’s been pointed out that if they could use our test suites that would save them an enormous amount of development – even if they re-implement from scratch. All that said if the gtest usage stays relatively simple the conversion won’t likely be that difficult.
Call it what you want, but it’s an opportunity to improve the ecosystem by making the job of the few people in the world that provide implementations easier. This of course was expressed by one of these people in 2024 at c++now which led to the long conversation about Lit, etc.
All that said, if people really want to use gtest that’s ok with me, I just personally don’t want to use it. In my view it’s wildly over-complicated for testing these sorts of components - meaning fundamental libraries that are targeting the standard. Also a big set of tests these days should be running in constant evaluation which basically uses static_assert for tests. So I’m against having it as the default, but that’s it.
Of course, maybe in 2025 it’s all moot mostly – feed some tests to an AI and it will probably convert them from one framework to another pretty easily…
It’s a worthy cause, no doubt, but I’m much more motivated to improve the jobs of millions by upping the quality of C++ standard libraries.
Of course, you could argue that the millions will have improved lives by getting access to C++ standard libraries sooner if we use a testing framework that the standard library implementors like, but, yeah, AI or something .