Transition to VM based CI

[Ben Cooksley]

On Tue, Jul 8, 2025 at 7:04 AM Soumyadeep Ghosh <
soumyadeepghosh2004 at zohomail.in> wrote:

> Hello again,
>

Hi Soumyadeep,

> So, the last time I checked the notary service, what it does is basically
> sends the artifact to a remote server, where the artifact is signed with a
> valid key that's recognisable as KDE's to validate the artifact as KDE's
> artifact in various stores. Am I explaining it correctly?
>

That is only part of what it does.

The Notary Service exists to provide a secure and protected way to do
privileged operations on the CI system. The Notary Service exists not only
to handle using signing keys (some of which will in the future be in a HSM
due to mandated requirements for Windows signing keys), but also anything
else which involves sensitive materials on CI - like API keys. This is how
the Google Play and Microsoft Store publishers work.

It allows us to enforce validation checks (to ensure that an application
only publishes under names it is allowed to for instance) as part of it's
operation which includes ensuring certain authentication and authorization
checks are met as a default position.

>
> If I am right, then, I don't think that's needed for snaps. To upload a
> snap to the store, one basically needs the auth token of the KDE account in
> the snap store and pass it when uploading the snap. IIRC, till now, some of
> the vms were having this auth token embedded. Can't we replicate something
> like this?
>

That is an inherently insecure approach and is why publishing only worked
for certain repositories within invent.kde.org/neon/.

It may be the path of least resistance, but it offers no validation or
other checks and also exposes the sensitive credential (the API key) to the
CI job - allowing for it to be used to do things other than publishing
applications.
A Notary Service prevents this from happening and helps secure the build
pipeline process and is why a Notary is required to implement publishing in
production.


> Thanks,
>
> Soumyadeep
>

Regards,
Ben

> On 08/07/25 00:20, Ben Cooksley wrote:
>
> On Mon, Jul 7, 2025 at 11:55 PM Soumyadeep Ghosh <
> soumyadeepghosh2004 at zohomail.in> wrote:
>
>> Hi Ben,
>>
>
> Hi Soumyadeep,
>
>
>>
>> This is an awesome news! I just wanted to know what would be the problem
>> in publishing those snaps built inside the new VM based CI.
>>
>
> There is currently no Notary built that supports handling the publishing
> process at this time, which is why we're unable to publish.
>
>
>>
>> Thanks,
>> Soumyadeep Ghosh
>>
>
> Cheers,
> Ben
>
>
>>
>>
>>
>>
>> ---- On Mon, 07 Jul 2025 17:19:14 +0530 * bcooksley at kde.org
>> <bcooksley at kde.org> * wrote ----
>>
>> Hi all,
>>
>> I'm happy to announce that VM based CI is now in a state where it is
>> ready to begin making the transition into general availability. As part of
>> this i'm happy to advise that builds for Snaps (but not their publishing)
>> are also becoming generally available, and FreeBSD will be updating to Qt
>> 6.9 as well.
>>
>> This transition will also mark the general retirement of Docker based CI,
>> with Docker functionality only being retained for website builds and linter
>> jobs (such as xml-lint, cppcheck, etc).
>>
>> As part of this support for Qt 5 (except for general Linux) is also being
>> retired. This includes all FreeBSD and Windows support, as well as all
>> binary support (Appimages, Windows exes, etc)
>>
>> *Steps you need to take:*
>>
>> If you only inherit the templates from sysadmin/ci-utilities, and do not
>> have custom jobs, then no action is needed on your part.
>>
>> If you have custom jobs which are based on the templates currently in
>> sysadmin/ci-utilities then  you should examine the diff in
>> https://invent.kde.org/sysadmin/ci-utilities/-/merge_requests/507 to
>> ensure that no changes are needed to your job.
>>
>> Custom jobs not based on the sysadmin provided templates likely do not
>> need to be changed.
>>
>> Projects that are Qt 5 based should remove includes for everything except
>> Linux general (linux.yml) as those templates will be removed as part of
>> this roll out.
>>
>> *Timeline for the transition:*
>>
>> Over the next week the first node (node6.ci.kde.org) will be taken out
>> of rotation for running existing container based jobs to facilitate testing
>> of the automated deployment. Once this has been validated, the conversion
>> of the remaining nodes (node2, node3, node4, node5) will be scheduled -
>> converting them all to be VM based CI runners.
>>
>> To allow for final legacy jobs to be completed, node1 will be left to run
>> remaining legacy jobs for a period of time (to be determined). Once that
>> finishes up, that node will also be converted.
>>
>> As part of this the dedicated VM runners currently supplied for building
>> KDE Linux and Snaps will also be retired. Support for Yocto builds is out
>> of scope for this transition at this time but that may be re-evaluated in
>> the future.
>>
>> *What specs will the VMs be provided with?*
>>
>> By default builds will be provided with 8 vCPU cores, 16GB RAM and 200GB
>> of disk space, although some of that disk space will be occupied by the VM
>> OS, development tools, etc.
>> Builds requiring additional resources should file a Sysadmin ticket.
>>
>> VMs will have a shared mount provided from the host at /mnt that allows
>> for artifacts to be cached between builds on that node. This is mainly
>> intended to be used for caching dependencies and other downloaded artifacts
>> and should not be relied on to transfer results between jobs (as there is
>> no guarantee contents won't go away, or that jobs will be allocated to the
>> same build node).
>>
>> *What benefits will we get as part of this change?*
>>
>> For Linux based builds, as these are now running in a fully fledged
>> system, anything that depends on system level services (like CUPS or
>> systemd/logind) or which needs to undertake actions normally restricted in
>> a Docker/Podman container (like manipulating FUSE mounts or interacting
>> with hardware like a TPM) should now be able to be unit tested.
>>
>> For FreeBSD based builds, these images no longer have to be built
>> manually by hand, and are now easily available to be run locally on
>> developer systems for local testing. Reliability of FreeBSD builds should
>> also improve with the elimination of the dependency on the Podman daemon
>> being running on the build node.
>>
>> For Windows based builds, these should run faster due to the elimination
>> of the overhead of the Docker overlay file system, which on Windows
>> introduces significant overhead to IO operations. For Kate, regular Windows
>> CI build times were reduced from 6 minutes (under Docker) to 3 minutes 40
>> seconds (in a VM based CI setup). This includes the overhead of
>> provisioning the VM and waiting for Windows to start. We should also see
>> improvements to builder availability, as the out of memory issues that have
>> caused Windows build nodes to be killed by the OOM-killer are not possible
>> in a VM based CI environment.
>>
>> For binary builds, Linux support is enhanced by it now being possible to
>> build Snaps, as well as Flatpaks no longer requiring workarounds to be
>> built, while Appimages are now able to be run if needed. These are actions
>> that have all previously been restricted by operating in a container
>> environment.
>>
>> The system will also generally benefit from being able to scale build
>> capacity for different OSes on an as needed basis within our available
>> hardware (so long build queues for Windows post major release events should
>> become less of an issue).
>>
>> As an additional benefit, the system will require significantly less work
>> to maintain. Currently each build node, along with the FreeBSD and Windows
>> VM thereon, have to be maintained by hand and disk space allocated between
>> them in a fixed fashion. This means that any cleanup from stale disk
>> images, over-filled caches, etc. has to be done 3 times on each build node
>> (being the Linux host as well as the FreeBSD and Windows guest VMs).
>> Currently provisioning new nodes is significantly labour intensive as well
>> (see
>> https://invent.kde.org/sysadmin/ci-utilities/-/blob/master/gitlab-templates/README.md
>> for the instructions).
>>
>> This is essentially completely eliminated with the transition to VM based
>> CI, with the majority of the deployment now being possible using Ansible
>> with the only manual step being the registration with Gitlab - which is a
>> fairly quick process taking less than 20 minutes per node. Maintenance is
>> significantly reduced as each node only needs one set of cleanup - not
>> three.
>>
>> Should there be any questions on the above please let me know.
>>
>> Many thanks,
>> Ben
>>
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.kde.org/pipermail/plasma-devel/attachments/20250708/a34d9e8f/attachment-0001.htm>