Re: Transition to VM based CI

Mon Jul 7 12:55:05 BST 2025

Hi Ben,

This is an awesome news! I just wanted to know what would be the problem in publishing those snaps built inside the new VM based CI.

Thanks,
Soumyadeep Ghosh

 ---- On Mon, 07 Jul 2025 17:19:14 +0530  bcooksley at kde.org  wrote ----

Hi all,

I'm happy to announce that VM based CI is now in a state where it is ready to begin making the transition into general availability. As part of this i'm happy to advise that builds for Snaps (but not their publishing) are also becoming generally available, and FreeBSD will be updating to Qt 6.9 as well.

This transition will also mark the general retirement of Docker based CI, with Docker functionality only being retained for website builds and linter jobs (such as xml-lint, cppcheck, etc). 

As part of this support for Qt 5 (except for general Linux) is also being retired. This includes all FreeBSD and Windows support, as well as all binary support (Appimages, Windows exes, etc)

Steps you need to take:

If you only inherit the templates from sysadmin/ci-utilities, and do not have custom jobs, then no action is needed on your part.

If you have custom jobs which are based on the templates currently in sysadmin/ci-utilities then  you should examine the diff in  https://invent.kde.org/sysadmin/ci-utilities/-/merge_requests/507  to ensure that no changes are needed to your job.

Custom jobs not based on the sysadmin provided templates likely do not need to be changed.

Projects that are Qt 5 based should remove includes for everything except Linux general (linux.yml) as those templates will be removed as part of this roll out.

Timeline for the transition:

Over the next week the first node ( http://node6.ci.kde.org ) will be taken out of rotation for running existing container based jobs to facilitate testing of the automated deployment. Once this has been validated, the conversion of the remaining nodes (node2, node3, node4, node5) will be scheduled - converting them all to be VM based CI runners.

To allow for final legacy jobs to be completed, node1 will be left to run remaining legacy jobs for a period of time (to be determined). Once that finishes up, that node will also be converted.

As part of this the dedicated VM runners currently supplied for building KDE Linux and Snaps will also be retired. Support for Yocto builds is out of scope for this transition at this time but that may be re-evaluated in the future.

What specs will the VMs be provided with?

By default builds will be provided with 8 vCPU cores, 16GB RAM and 200GB of disk space, although some of that disk space will be occupied by the VM OS, development tools, etc. 
Builds requiring additional resources should file a Sysadmin ticket.

VMs will have a shared mount provided from the host at /mnt that allows for artifacts to be cached between builds on that node. This is mainly intended to be used for caching dependencies and other downloaded artifacts and should not be relied on to transfer results between jobs (as there is no guarantee contents won't go away, or that jobs will be allocated to the same build node).

What benefits will we get as part of this change?

For Linux based builds, as these are now running in a fully fledged system, anything that depends on system level services (like CUPS or systemd/logind) or which needs to undertake actions normally restricted in a Docker/Podman container (like manipulating FUSE mounts or interacting with hardware like a TPM) should now be able to be unit tested.

For FreeBSD based builds, these images no longer have to be built manually by hand, and are now easily available to be run locally on developer systems for local testing. Reliability of FreeBSD builds should also improve with the elimination of the dependency on the Podman daemon being running on the build node.

For Windows based builds, these should run faster due to the elimination of the overhead of the Docker overlay file system, which on Windows introduces significant overhead to IO operations. For Kate, regular Windows CI build times were reduced from 6 minutes (under Docker) to 3 minutes 40 seconds (in a VM based CI setup). This includes the overhead of provisioning the VM and waiting for Windows to start. We should also see improvements to builder availability, as the out of memory issues that have caused Windows build nodes to be killed by the OOM-killer are not possible in a VM based CI environment.

For binary builds, Linux support is enhanced by it now being possible to build Snaps, as well as Flatpaks no longer requiring workarounds to be built, while Appimages are now able to be run if needed. These are actions that have all previously been restricted by operating in a container environment.

The system will also generally benefit from being able to scale build capacity for different OSes on an as needed basis within our available hardware (so long build queues for Windows post major release events should become less of an issue).

As an additional benefit, the system will require significantly less work to maintain. Currently each build node, along with the FreeBSD and Windows VM thereon, have to be maintained by hand and disk space allocated between them in a fixed fashion. This means that any cleanup from stale disk images, over-filled caches, etc. has to be done 3 times on each build node (being the Linux host as well as the FreeBSD and Windows guest VMs). Currently provisioning new nodes is significantly labour intensive as well (see  https://invent.kde.org/sysadmin/ci-utilities/-/blob/master/gitlab-templates/README.md  for the instructions). 

This is essentially completely eliminated with the transition to VM based CI, with the majority of the deployment now being possible using Ansible with the only manual step being the registration with Gitlab - which is a fairly quick process taking less than 20 minutes per node. Maintenance is significantly reduced as each node only needs one set of cleanup - not three.

Should there be any questions on the above please let me know.

Many thanks,
Ben
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.kde.org/pipermail/release-team/attachments/20250707/d228266e/attachment-0001.htm>