Upcoming CI changes - transition to VM based CI

[Albert Astals Cid]

El dilluns, 2 de juny del 2025, a les 16:42:01 (Hora d’estiu d’Europa central), Christoph Cullmann va escriure:
> Hi,
> 
> On Monday, June 2nd, 2025 at 13:40, Ben Cooksley <bcooksley at kde.org> wrote:
> > Hi all,
> > For some time now we have had a variety of issues with our Docker/Podman
> > based CI builds. These have included the lack of GUI test support on
> > Windows, periodic crashes on FreeBSD, poor IO performance of Windows
> > builds, issues supporting builds for Flatpak and Snaps and inability to
> > support either builds or tests where elevated privileges or system
> > session resources are needed.
> > 
> > 
> > In addition to this we've had issues where Linux CI builds have the
> > capability to trigger OOM events on the CI hosts, which in turn takes out
> > Windows and (less often) FreeBSD builders. While this does not occur too
> > often, it does happen from time to time and eventually negatively impacts
> > the build queue for those platforms.
> > 
> > 
> > The need to have dedicated VMs for FreeBSD and Windows on our builders
> > also makes setting up of a CI build node for KDE software a more
> > complicated and time intensive task than it otherwise needs to be (and
> > means that the amount of systems to care for increases by 3 for every CI
> > node we add).
> > 
> > 
> > While individually relatively minor, together these issues more than
> > justify making a significant change to the way we run our CI system - in
> > this case transitioning from container based builds to VM based builds.
> > 
> > 
> > These builds will still take place on dedicated hardware that we control,
> > however instead of taking place within a container (managed by Podman on
> > Linux and FreeBSD, or Docker on Windows) they will instead take place
> > within a VM using a copy-on-write disk image. VM based builds will
> > unfortunately take a little longer to start (it takes ~10 seconds for a
> > VM from any of Linux, FreeBSD or Windows to boot on my personal system)
> > however the benefits we gain should more than outweigh this small
> > downside.
> > 
> > 
> > This has been under development for the past couple of weeks and is now
> > reaching the point where the only remaining steps are to get it
> > integrated with the Gitlab CI agent (gitlab-runner) for which prototype
> > code is already in place, and complete porting of our images over. Once
> > that happens a complete rebuild of all of our builders will be swiftly
> > undertaken to transition them completely over to the new VM based
> > infrastructure.
> > 
> > 
> > Specs wise, at this time it is planned for each spawned standard VM to be
> > provided with 2/3's of the system CPU cores (so 12 cores), 16GB RAM and
> > 100GB of disk space (although some of that will be occupied by the system
> > image - approximately 10GB for standard Linux builds and ~30GB or so for
> > Windows builds). There will be a higher resource tier available for
> > certain builds however that will be on request only and would need to be
> > justified (such as Craft needing to build QtWebEngine).
> > 
> > 
> > As launching VMs is not the most efficient approach for all workloads,
> > limited support for running Docker containers will be preserved, however
> > this support is primarily intended for running linters, sanity checks and
> > website builds, and is not intended for running general CI/CD builds.
> > 
> > 
> > The tooling used by the CI nodes to run VMs is something that should be
> > fairly trivial for people to run on their own local system should they
> > wish to run any of those images (say for FreeBSD or Android), although
> > you will need to setup libvirt yourself (SUSE has very good instructions
> > for this, Debian less so as their instructions lack installing the
> > packages needed to provide UEFI and TPM support). The tooling itself was
> > merged this evening to sysadmin/ci-images (vm-common/ folder) and can be
> > used with the VM images found at https://storage.kde.org/vm-images/
> > 
> > 
> > There is however one downside to this - Qt 5 support.
> > 
> > 
> > Over the past few months distributions have been steadily removing
> > packages and other supporting infrastructure needed to keep Qt 5 builds
> > alive. In the case of Windows, support for the entire Qt 5 tree has been
> > unmaintained for some time. For FreeBSD and SUSE a significant number of
> > packages have been removed - which in the case of SUSE also includes
> > packages needed to support the building of KJS. Accordingly, because
> > builds of Frameworks are a first stepping stone to support building
> > anything else, it will not be possible for us to produce Qt 5 based VM
> > build images for any of the 3 platforms.
> > 
> > 
> > We will therefore have to remove Qt 5 support from the CI system with the
> > transition to VM based CI.
> > 
> > 
> > Please let me know if there are any questions on the above.
> 
> Have we some overview how many things on invent.kde.org will loose the the
> CI as they are still Qt 5 only?

Here you have a list of what scripty considers KF5 in master

https://invent.kde.org/sysadmin/l10n-scripty/-/blob/master/get_paths.trunk_kf5?ref_type=heads#L84

There's some things that are not really KF5 like webpages or projects without UI, but the list is non-neglegible and it has quite a few KDE Gear applications.

Cheers,
  Albert

> 
> Greetings
> Christoph