My strong preference is to make a Beman requirement/recommendation and to have uniform approach everywhere.
I see few aspects here:
We should have the same set of images to run on CI for all repos (e.g. image with ubuntu-24 and a single compiler installed - clang-18 → image == ubuntu-24:clang-18). I don’t know if we need to create Docker files or we can have a set of CI images customized by use.
These images (e.g. Docker) should not be required for local development. Our build instructions should work directly on host.
We should be able to use the CI images on our local systems if we want. It’s hard to install a matrix of compilers and not get weird scenarios (e.g., optional26 only compiles with clang-17, but newer libc++ version - 18 or 19 - on my VM with ubuntu). If somebody managed to create an image with a stable environment, why not sharing with others?
Can Docker images help with portability? e.g., if I have a Docker file with ubuntu24 x86_64 image, I can run on arm64 machines with an extra parameter (e.g. optional26/docs/debug-ci.md at main · beman-project/optional26 · GitHub). This is sometimes slow… Does anybody have improvement suggestion here?
I don’t think we need to be using docker in containers for CI. Docker-ception is a bit of an anti-pattern?
I see docker as primarily useful for local development, shipping a working and consistent development environment. Running it as part of CI is then mostly a check of the environment, more than the software under test.
It is, however, easier to maintain an environment within a container. For example, the LLVM apt repo marks the runtime support libraries as implementing the same feature, so you only have the latest cxxabi library, and this means occasional differences between having both LLVM 18 and 19, and just 18. On the other hand, having everyone build and install multiple LLVM versions themselves is infeasible.
Supporting Docker development (as opposed to allowing) is a bit difficult, though, without also mandating a whole bunch of specific tools and extensions to those tools.
An ecosystem wide problem is that there’s no agreed on way of describing, in code, a project, it’s dependencies, build options, tools. Python has pyproject.toml, for example. Python projects are much more portable across developers systems than C++ projects. Other ecosystems, like Rust and Go, have much lower variation, so there’s also more portability.
They launch really quickly (< 5s) and are usually zero-pain with minimal maintenance. Build and test can be done in ~30s for all matrix.
GitHub action have a pretty good eco-system, we don’t have to worry about setting-up tools across platform to a specific version, there’s usually a GitHub Actions Package for that (e.g. cmake, pre-commit).
In comparison, building and spinning up the docker environment alone takes ~30s, we will need to maintain our own docker images, our own setup scripts across platforms, and testing on macos could be more painful. Plus, GitHub action packages doesn’t work if we need to launch the image with docker command (and is not desirable).
The upside of using docker is we can test across platforms locally, and the os would be more clean due to the absence of all the extra developer tools.
I recall there was an option to SSH into the action runner for debuging but can no-longer find the documentation for that
I’m supportive of ditching docker in favor of the GitHub CI images. Docker is only Linux and we need to have good stories for MacOS and Windows.
If a particular Beman project has a need for a specialized docker container, they should go-ahead of course, but I think the default (beman.exemplar) should be GitHub CI images.
I’m flexible as long as we’re testing what we need to and the developer experience is good.
I’ll reiterate that we should be cautious about requiring things that we can’t automatically detect or even maintain.
I could see a recommendation that lends support to relevant PRs. I would like to see some engineering work to automate things before making this sort of thing a requirement.