Code coverage with coveralls

Beman Project Development
1

In beman.utf_view, we get code coverage using gcov and Coveralls.

utf_view currently has three CI jobs, configured using GitHub Actions: gcc-linux, clang-linux, and msvc-windows. Coverage is currently measured as part of the gcc-linux CI job.

To ensure that running the unit tests generates gcov data, the gcc-linux CI job sets -DCMAKE_CXX_FLAGS='-coverage' when compiling.

It then runs the unit tests using ctest.

Note that all the functionality utf_view proposes for standardization is constexpr, so most of the unit tests are implemented as constexpr functions returning a bool. These unit tests can be checked by invoking them inside a static_assert, but to ensure we get coverage data, utf_view makes sure to also execute the test functions at runtime.

After the unit tests have generated the raw gcov coverage data, an additional lcov step in the CI job consolidates the gcov data into a .info file:

lcov --directory ./build --capture --output-file ./build/coverage_all.info

It then removes any coverage data originating from the standard library, thirdparty dependencies, or the unit tests themselves:

lcov --remove ./build/coverage_all.info -o ./build/coverage.info '/usr/include/*' "$PWD/tests/beman/utf_view/*" "$PWD/build/_deps/*"

Finally, the filtered test coverage file is uploaded to Coveralls using their GitHub Action.

Coveralls provides a page for each source file showing covered lines in green and uncovered lines in red. An example is here, although you need to log in to Coveralls to see it (otherwise it will say “Source Not Available”): bemanproject/utf_view | Build 11933026153 | include/beman/utf_view/to_utf_view.hpp | Coveralls - Test Coverage History & Statistics

Coveralls automatically leaves comments on pull requests with coverage reports: Add test random access iterator and corresponding concept check for code unit view by ednolan · Pull Request #14 · bemanproject/utf_view · GitHub

And it provides a coverage badge that can be added to the README file.

This approach does not require any modification to the CMake, since users can simply provide -coverage using -DCMAKE_CXX_FLAGS.

To my understanding, although my current implementation in utf_view uses Docker, this isn’t a hard requirement, since lcov can be installed onto a GitHub runner image.

It doesn’t cost anything because Coveralls only charges for private repos, stating “Coveralls will always be free for open source.”

I’m interested in feedback about this approach-- if we think we might want to roll it out to other repositories, I can create a pull request for Optional26 to add coverage to that repository as a first step.

3 Likes
2

Makes sense to me. Nice writeup!

1 Like
3

It ought to be straightforward for Optional26 as I’ve got some support for gcov local reporting in the repo already, but not running through CI.

4

Coveralls support for optional26 has been merged.

5

Thanks for the write up.

I am interested in porting this to exemplar and I love how clean the setup is.

CI side, I will probably merge gcc-linux, clang-linux, and msvc-windows into a single matrix-driven action. So we can inline ci.sh , reduce the need to jump around the project to understand the CI system.

But I will separate out coverage generation from main testing jobs, it is too much of an exception that I feel like it would easily clutter the main testing matrix and deserve to be its own job. I think your PR to optional26 to implement coverage reporting is a good example of this happening, multiple CI steps only getting executed on one variant of the matrix combination, requiring an extra parameter to the whole matrix that only gets invoked once. It may become unclear once the matrix gets more complicated if there’s only one testing matrix that produce and uploads the coverage data. Though to be fair, a separate coverage generation job will leave room for the testing parameters to go out of sync with each other.

I will implement code coverage report with inspiration from your write-up as its own separate job, then we can see if this is a good trade-off.

I do have a question, why are we using lcov here and stripping third party dependencies, does coveralls not work when there’s coverage data outside of the source tree? I don’t see there being a script to generate this coverage data on the developer side, thus it looks like the only consumer for the gocv data is coveralls , I wonder lcov and the stripping step could be omitted.

Agh I see why you use lcov, gcov is painful without cmake support/ using lcov.

I do wonder how will we be able to handle cases where there are multiple execution path depending on the platform, we will need to merge all the coverage data together to have a more coherent view of the whole coverage situation.

1 Like
6

But I will separate out coverage generation from main testing jobs, it is too much of an exception that I feel like it would easily clutter the main testing matrix and deserve to be its own job. I think your PR to optional26 to implement coverage reporting is a good example of this happening, multiple CI steps only getting executed on one variant of the matrix combination, requiring an extra parameter to the whole matrix that only gets invoked once.

That sounds good to me. I was on the fence about which approach was better, and originally my optional26 commit added coverage steps outside of the matrix. I have no objection to making it its own job.

does coveralls not work when there’s coverage data outside of the source tree?

The way the gcov data is generated is by providing a -coverage flag to GCC, which then ensures that when an executable is run, it generates coverage data for every file that it has associated debuginfo for. This includes, for example, files in the standard library. If you don’t filter them out, they affect the coverage score.

Here’s an example of a build from when I was working on utf_view before I figured out filtering, which is complaining that I don’t fully cover the <new> header: ednolan/UtfView | Build 10541310543 | ../../../usr/include/c++/14/new | Coveralls - Test Coverage History & Statistics

beman.optional26 implements this filtering via gcovr configuration: optional26/cmake/gcovr.cfg.in at 183e933784ef67954c2513aaa38232841b4a0b81 · bemanproject/optional26 · GitHub

I don’t see there being a script to generate this coverage data on the developer side, thus it looks like the only consumer for the gcov data is coveralls

The other consumer is me-- except that instead of using a script, I was just running lcov manually, so I could look at the html files locally. The experience with optional26 is nicer because it has a special Gcov build configuration so you don’t have to remember how to do that manually.

I wonder lcov and the stripping step could be omitted.

Some form of stripping is still needed because you have to have some way of telling coveralls that e.g., the standard library includes shouldn’t count towards the coverage score.

But it might be possible to eliminate lcov. When I initially looked at this, the coveralls GitHub action seemed to only accept a configuration file path-to-lcov, so I used lcov for that reason. But then later when working on optional26, I checked the documentation and it said that path-to-lcov was deprecated and replaced by file and it accepted a large variety of formats, including directly accepting gcov files. It turned out the reason I missed this was that I had set my GitHub Action to use @master instead of @main :person_facepalming:.

So it’s very possible that with some more work on configuring Coveralls we can eliminate the lcov step in utf_view. But we might still want to keep it, for the same reason we have an intermediate gcovr step in optional26, which is that it provides a way for users to inspect the coverage results locally.

I do wonder how will we be able to handle cases where there are multiple execution path depending on the platform, we will need to merge all the coverage data together to have a more coherent view of the whole coverage situation.

I wonder this too. I don’t know how we’d address that problem.

7

When I checkout the big standard library implementations, they use reusable workflows to accomplish this, basically you have composite based action package somewhere and you write it like writing a GitHub Action package. Invocation looks something like:

- name: Run Tests
  uses: .github/utils/test-action

It is a bit too complicated for beman.

I am surprised coveralls don’t do any normalization for us. But I think you are right.

When I run coverall locally, it seems to be able to scan for gcov output directly. However, the invocation of gcov is fairly painful, it asks for the path of .gcda files. A script will be needed to work with gcov, either in cmake or shell. Using lcov here is a good idea.

There is a parallel option on coveralls 's GitHub Actions. I can try to play around with that to see what it does.

8

Playing with coveralls myself, I would suggest you replace the stripping command with:

lcov --extract coverage_all.info -o coverage.info "$PWD/include/*" "$PWD/src/*"

Instead of --remove I think it would be a good idea to use --include. Given we have a defined project structure, we should be able to assume device under test should reside under those two directories.

9

Maybe we shouldn’t use lcov.

There seem not to be a good lcov alliterative for windows. We might want collect coverage from MSVC based tool chains so we probably should use alternatives like gcovr.

See: gcov - Is there any actively supported lcov port for windows - Stack Overflow

1 Like
10

I use gcovr python module in the past:

bash-5.2$ cat gcovr.cfg
root = .
search-path = build

filter = source/*
filter = include/*
filter = standalone/*
# filter = build/all/_dpes/greeter-build/*

exclude-directories = test
exclude-directories = stagedir
exclude-directories = .cache

gcov-ignore-parse-errors = all
print-summary = yes

html-details = build/gcovr.html

cobertura-pretty = yes
cobertura = build/cobertura.xml

bash-5.2$ gcovr 
(INFO) Reading coverage data...
(INFO) Writing coverage report...
lines: 100.0% (34 out of 34)
functions: 100.0% (4 out of 4)
branches: 52.0% (91 out of 175)
bash-5.2$
11

No objection from me to standardizing on gcovr.

1 Like