Transition to VM based CI

[Ben Cooksley]

Hi all,

I'm happy to announce that VM based CI is now in a state where it is ready
to begin making the transition into general availability. As part of this
i'm happy to advise that builds for Snaps (but not their publishing) are
also becoming generally available, and FreeBSD will be updating to Qt 6.9
as well.

This transition will also mark the general retirement of Docker based CI,
with Docker functionality only being retained for website builds and linter
jobs (such as xml-lint, cppcheck, etc).

As part of this support for Qt 5 (except for general Linux) is also being
retired. This includes all FreeBSD and Windows support, as well as all
binary support (Appimages, Windows exes, etc)

*Steps you need to take:*

If you only inherit the templates from sysadmin/ci-utilities, and do not
have custom jobs, then no action is needed on your part.

If you have custom jobs which are based on the templates currently in
sysadmin/ci-utilities then  you should examine the diff in
https://invent.kde.org/sysadmin/ci-utilities/-/merge_requests/507 to ensure
that no changes are needed to your job.

Custom jobs not based on the sysadmin provided templates likely do not need
to be changed.

Projects that are Qt 5 based should remove includes for everything except
Linux general (linux.yml) as those templates will be removed as part of
this roll out.

*Timeline for the transition:*

Over the next week the first node (node6.ci.kde.org) will be taken out of
rotation for running existing container based jobs to facilitate testing of
the automated deployment. Once this has been validated, the conversion of
the remaining nodes (node2, node3, node4, node5) will be scheduled -
converting them all to be VM based CI runners.

To allow for final legacy jobs to be completed, node1 will be left to run
remaining legacy jobs for a period of time (to be determined). Once that
finishes up, that node will also be converted.

As part of this the dedicated VM runners currently supplied for building
KDE Linux and Snaps will also be retired. Support for Yocto builds is out
of scope for this transition at this time but that may be re-evaluated in
the future.

*What specs will the VMs be provided with?*

By default builds will be provided with 8 vCPU cores, 16GB RAM and 200GB of
disk space, although some of that disk space will be occupied by the VM OS,
development tools, etc.
Builds requiring additional resources should file a Sysadmin ticket.

VMs will have a shared mount provided from the host at /mnt that allows for
artifacts to be cached between builds on that node. This is mainly intended
to be used for caching dependencies and other downloaded artifacts and
should not be relied on to transfer results between jobs (as there is no
guarantee contents won't go away, or that jobs will be allocated to the
same build node).

*What benefits will we get as part of this change?*

For Linux based builds, as these are now running in a fully fledged system,
anything that depends on system level services (like CUPS or
systemd/logind) or which needs to undertake actions normally restricted in
a Docker/Podman container (like manipulating FUSE mounts or interacting
with hardware like a TPM) should now be able to be unit tested.

For FreeBSD based builds, these images no longer have to be built manually
by hand, and are now easily available to be run locally on developer
systems for local testing. Reliability of FreeBSD builds should also
improve with the elimination of the dependency on the Podman daemon being
running on the build node.

For Windows based builds, these should run faster due to the elimination of
the overhead of the Docker overlay file system, which on Windows introduces
significant overhead to IO operations. For Kate, regular Windows CI build
times were reduced from 6 minutes (under Docker) to 3 minutes 40 seconds
(in a VM based CI setup). This includes the overhead of provisioning the VM
and waiting for Windows to start. We should also see improvements to
builder availability, as the out of memory issues that have caused Windows
build nodes to be killed by the OOM-killer are not possible in a VM based
CI environment.

For binary builds, Linux support is enhanced by it now being possible to
build Snaps, as well as Flatpaks no longer requiring workarounds to be
built, while Appimages are now able to be run if needed. These are actions
that have all previously been restricted by operating in a container
environment.

The system will also generally benefit from being able to scale build
capacity for different OSes on an as needed basis within our available
hardware (so long build queues for Windows post major release events should
become less of an issue).

As an additional benefit, the system will require significantly less work
to maintain. Currently each build node, along with the FreeBSD and Windows
VM thereon, have to be maintained by hand and disk space allocated between
them in a fixed fashion. This means that any cleanup from stale disk
images, over-filled caches, etc. has to be done 3 times on each build node
(being the Linux host as well as the FreeBSD and Windows guest VMs).
Currently provisioning new nodes is significantly labour intensive as well
(see
https://invent.kde.org/sysadmin/ci-utilities/-/blob/master/gitlab-templates/README.md
for the instructions).

This is essentially completely eliminated with the transition to VM based
CI, with the majority of the deployment now being possible using Ansible
with the only manual step being the registration with Gitlab - which is a
fairly quick process taking less than 20 minutes per node. Maintenance is
significantly reduced as each node only needs one set of cleanup - not
three.

Should there be any questions on the above please let me know.

Many thanks,
Ben
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.kde.org/pipermail/release-team/attachments/20250707/dd64b8ab/attachment.htm>