Bases of suggestion
There is a common desire of testing library across multiple compiler versions.
e.g. Enable latest versions of compilers on main · Issue #73 · bemanproject/exemplar · GitHub / attempt in option26
This is needed but sometimes prohibitive due to setup complexity (there’s numerous matrixs like this GitHub - egor-tensin/setup-gcc: GitHub action to set up GCC that shows what compiler version is easily available on each platform). Sometimes apt-install takes care of the job, sometimes you need a specific ppa, sometimes you have to install via the release page, sometimes you have to build it.
Install scripts often needs to source the build from multiple places (GitHub, PPA) that creates uncertainty to job execution time and increases unnecessary fault.
GitHub Actions community does not have a well maintained setup helper for compilers that minimizes setup time.
Attempts as listed:
- GitHub - egor-tensin/setup-gcc: GitHub action to set up GCC (maybe abandoned)
- GitHub - KyleMayes/install-llvm-action: A GitHub Action for downloading and installing LLVM and Clang binaries. (does not utilize caching thus it takes same time to install as using the official install script)
Suggestion
It might be beneficial for us to have a pre-built docker image hosted on GitHub to ensure common compiler-test environment. We should have its script hosted in the new infrastructure repo.
Ideally we should be able to test across compiler versions by just setting the CI job image to e.g. beman/cpp-image:gnu-14 provides gcc-14 compiler. Cross-compiler test should be as easy as “run on this image (e.g. llvm-17), configure and test”.
Non-goal
I don’t think we should move all CI jobs to our specific images. Running linters and basic testing should still be in the default GitHub provided environment. The default images are fast to setup, well-maintained, feature-rich, and provides good cross-platform support.
I don’t think this image should be used for personal development use, e.g. putting in developer tools such as language servers, vscode, fancy terminals. It does not attempt to resolve the “it works on my machine” from the developer side. Though it definitely provides the infrastructure for a better devcontainer image.
Example of this simplifying CI jobs
Before:
compiler-test:
runs-on: ubuntu-24.04
strategy:
matrix:
compilers:
- class: GNU
version: 14
- class: GNU
version: 13
- class: LLVM
version: 17
name: "Compiler: ${{ matrix.compilers.class }} ${{ matrix.compilers.version }}"
steps:
- uses: actions/checkout@v4
- name: Setup build environment
uses: lukka/get-cmake@latest
with:
cmakeVersion: "~3.25.0"
ninjaVersion: "^1.11.1"
- name: Install Compiler
id: install-compiler
run: |
if [ "${{ matrix.compilers.class }}" = "GNU" ]; then
CC=gcc-${{ matrix.compilers.version }}
CXX=g++-${{ matrix.compilers.version }}
sudo add-apt-repository universe
sudo apt-get update
sudo apt-get install -y $CC
sudo apt-get install -y $CXX
$CC --version
$CXX --version
else
wget https://apt.llvm.org/llvm.sh
chmod +x llvm.sh
sudo bash llvm.sh ${{ matrix.compilers.version }}
CC=clang-${{ matrix.compilers.version }}
CXX=clang++-${{ matrix.compilers.version }}
$CC --version
$CXX --version
fi
echo "CC=$CC" >> "$GITHUB_OUTPUT"
echo "CXX=$CXX" >> "$GITHUB_OUTPUT"
- name: Configure CMake
run: |
cmake -B build -S . -DCMAKE_CXX_STANDARD=20
env:
CC: ${{ steps.install-compiler.outputs.CC }}
CXX: ${{ steps.install-compiler.outputs.CXX }}
CMAKE_GENERATOR: "Ninja Multi-Config"
- name: Build Debug
run: |
cmake --build build --config Debug --verbose
cmake --build build --config Debug --target all_verify_interface_header_sets
cmake --install build --config Debug --prefix /opt/beman.exemplar
find /opt/beman.exemplar -type f
- name: Test Debug
run: ctest --test-dir build --build-config Debug
After:
compiler-test:
runs-on: ubuntu-24.04
strategy:
matrix:
image: ["gnu-14", "gnu-13", "llvm-17", "llvm-head"]
container:
image: ghcr.io/beman/cpp-docker:${{ matrix.image }}
name: "Compiler: ${{ matrix.image }}"
steps:
- uses: actions/checkout@v4
- name: Configure CMake
run: |
cmake -B build -S . -DCMAKE_CXX_STANDARD=20
env:
# consistent symlink at image build time from compiler (e.g. gcc-14) to
# common location e.g. /beman/compilers
CC: "/beman/compilers/c"
CXX: "/beman/compilers/cc"
CMAKE_GENERATOR: "Ninja Multi-Config"
- name: Build Debug
run: |
cmake --build build --config Debug --verbose
cmake --build build --config Debug --target all_verify_interface_header_sets
cmake --install build --config Debug --prefix /opt/beman.exemplar
find /opt/beman.exemplar -type f
- name: Test Debug
run: ctest --test-dir build --build-config Debug
Furtuer benefit
This would enable testing on HEAD of compilers/ branch (thus a PR) of a compiler that would be prohibitive right now due to the need to build the compiler at setup.
This would also enable us to switch to older versions of compilers that are hard to setup on newer images.
When we have the infrastructure up, we will be able to setup a more feature-rich devcontainer environment that contains all the compilers as requested by @neatudarius . Current devcontainer builds the container at setup, making this prohibitive.
Downside
We need to maintain setup script (1) and docker images (2), and have central repo (3) to host these.
Answers:
- We already will need to maintain setup instructions on the CI side, we are simply centralizing this script.
- Everything could be implemented solely in GitHub and automated with GitHub actions.
- I believe we are already working on a infrastructure repo.
Difference from previous discussion
Previous attempt of using docker in CI is different from this suggestion.
Previous attempt distributes a docker image build script to each repo and builds a docker image to run in for every CI Job. It was meant to alleviate “works on my machine” problem and provide a general developer facing environment, thus in principle each repo may have variation on the docker build script.
This suggestion propose we keep the common use case inside the generic GitHub runner image. It propose we use a pre-built docker container only when we attempt to execute a cross-compiler test, of which the facilities needed to setup such environment could be easily centralized. The overhead of maintaining docker image and subsequent build jobs can be easily amortized to each repo, and provide meaningful abstraction for a common test case. This works will right now as we seem to be hitting a infrastructure prorogation bottleneck, since we are sure we will have to host a centralized infrastructure repo.
Proof of concept
I’ve made a proof of concept in contrast to Compiler test by wusatosi · Pull Request #84 · bemanproject/exemplar · GitHub .
Docker build script needed to host such automation:
Docker image hosted on GitHub:
CI implementation:
The docker image is built as a mono-image that contains all desired compiler versions, so it is different from my proposal. But there is already a speedup for setting up LLVM family compilers.
Appearently pulling a 4GB image is faster than running the LLVM install script.