Re: Transition to VM based CI

[Ben Cooksley]

On Mon, Jul 7, 2025 at 11:55 PM Soumyadeep Ghosh <
soumyadeepghosh2004 at zohomail.in> wrote:

> Hi Ben,
>

Hi Soumyadeep,


>
> This is an awesome news! I just wanted to know what would be the problem
> in publishing those snaps built inside the new VM based CI.
>

There is currently no Notary built that supports handling the publishing
process at this time, which is why we're unable to publish.


>
> Thanks,
> Soumyadeep Ghosh
>

Cheers,
Ben


>
>
>
>
> ---- On Mon, 07 Jul 2025 17:19:14 +0530 * bcooksley at kde.org
> <bcooksley at kde.org> * wrote ----
>
> Hi all,
>
> I'm happy to announce that VM based CI is now in a state where it is ready
> to begin making the transition into general availability. As part of this
> i'm happy to advise that builds for Snaps (but not their publishing) are
> also becoming generally available, and FreeBSD will be updating to Qt 6.9
> as well.
>
> This transition will also mark the general retirement of Docker based CI,
> with Docker functionality only being retained for website builds and linter
> jobs (such as xml-lint, cppcheck, etc).
>
> As part of this support for Qt 5 (except for general Linux) is also being
> retired. This includes all FreeBSD and Windows support, as well as all
> binary support (Appimages, Windows exes, etc)
>
> *Steps you need to take:*
>
> If you only inherit the templates from sysadmin/ci-utilities, and do not
> have custom jobs, then no action is needed on your part.
>
> If you have custom jobs which are based on the templates currently in
> sysadmin/ci-utilities then  you should examine the diff in
> https://invent.kde.org/sysadmin/ci-utilities/-/merge_requests/507 to
> ensure that no changes are needed to your job.
>
> Custom jobs not based on the sysadmin provided templates likely do not
> need to be changed.
>
> Projects that are Qt 5 based should remove includes for everything except
> Linux general (linux.yml) as those templates will be removed as part of
> this roll out.
>
> *Timeline for the transition:*
>
> Over the next week the first node (node6.ci.kde.org) will be taken out of
> rotation for running existing container based jobs to facilitate testing of
> the automated deployment. Once this has been validated, the conversion of
> the remaining nodes (node2, node3, node4, node5) will be scheduled -
> converting them all to be VM based CI runners.
>
> To allow for final legacy jobs to be completed, node1 will be left to run
> remaining legacy jobs for a period of time (to be determined). Once that
> finishes up, that node will also be converted.
>
> As part of this the dedicated VM runners currently supplied for building
> KDE Linux and Snaps will also be retired. Support for Yocto builds is out
> of scope for this transition at this time but that may be re-evaluated in
> the future.
>
> *What specs will the VMs be provided with?*
>
> By default builds will be provided with 8 vCPU cores, 16GB RAM and 200GB
> of disk space, although some of that disk space will be occupied by the VM
> OS, development tools, etc.
> Builds requiring additional resources should file a Sysadmin ticket.
>
> VMs will have a shared mount provided from the host at /mnt that allows
> for artifacts to be cached between builds on that node. This is mainly
> intended to be used for caching dependencies and other downloaded artifacts
> and should not be relied on to transfer results between jobs (as there is
> no guarantee contents won't go away, or that jobs will be allocated to the
> same build node).
>
> *What benefits will we get as part of this change?*
>
> For Linux based builds, as these are now running in a fully fledged
> system, anything that depends on system level services (like CUPS or
> systemd/logind) or which needs to undertake actions normally restricted in
> a Docker/Podman container (like manipulating FUSE mounts or interacting
> with hardware like a TPM) should now be able to be unit tested.
>
> For FreeBSD based builds, these images no longer have to be built manually
> by hand, and are now easily available to be run locally on developer
> systems for local testing. Reliability of FreeBSD builds should also
> improve with the elimination of the dependency on the Podman daemon being
> running on the build node.
>
> For Windows based builds, these should run faster due to the elimination
> of the overhead of the Docker overlay file system, which on Windows
> introduces significant overhead to IO operations. For Kate, regular Windows
> CI build times were reduced from 6 minutes (under Docker) to 3 minutes 40
> seconds (in a VM based CI setup). This includes the overhead of
> provisioning the VM and waiting for Windows to start. We should also see
> improvements to builder availability, as the out of memory issues that have
> caused Windows build nodes to be killed by the OOM-killer are not possible
> in a VM based CI environment.
>
> For binary builds, Linux support is enhanced by it now being possible to
> build Snaps, as well as Flatpaks no longer requiring workarounds to be
> built, while Appimages are now able to be run if needed. These are actions
> that have all previously been restricted by operating in a container
> environment.
>
> The system will also generally benefit from being able to scale build
> capacity for different OSes on an as needed basis within our available
> hardware (so long build queues for Windows post major release events should
> become less of an issue).
>
> As an additional benefit, the system will require significantly less work
> to maintain. Currently each build node, along with the FreeBSD and Windows
> VM thereon, have to be maintained by hand and disk space allocated between
> them in a fixed fashion. This means that any cleanup from stale disk
> images, over-filled caches, etc. has to be done 3 times on each build node
> (being the Linux host as well as the FreeBSD and Windows guest VMs).
> Currently provisioning new nodes is significantly labour intensive as well
> (see
> https://invent.kde.org/sysadmin/ci-utilities/-/blob/master/gitlab-templates/README.md
> for the instructions).
>
> This is essentially completely eliminated with the transition to VM based
> CI, with the majority of the deployment now being possible using Ansible
> with the only manual step being the registration with Gitlab - which is a
> fairly quick process taking less than 20 minutes per node. Maintenance is
> significantly reduced as each node only needs one set of cleanup - not
> three.
>
> Should there be any questions on the above please let me know.
>
> Many thanks,
> Ben
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.kde.org/pipermail/release-team/attachments/20250708/866eb735/attachment.htm>