Upcoming CI changes - transition to VM based CI

[Ben Cooksley]

On Tue, Jun 3, 2025 at 11:04 AM Lukas Sommer <sommerluk at gmail.com> wrote:

> Hi.
>

Hi Lukas,


>
> I have a question. Currently we have a static Windows Qt5 build, but we
> don't have a static windows Qt6 build, right? So when removing Qt5, there
> won't be any static windows build at all anymore?
>

Unless someone does the work to look into what is needed to have Windows
static builds with Qt 6, that is correct. The changes needed may be rather
minimal.
I'm not entirely sure what is involved, and looking at the setup i'm not
sure if the builds are true static builds as I don't think we have a static
Qt build on Windows.


>
> Best regards
>
> Lukas
>

Thanks,
Ben


>
>
> Am 2. Juni 2025 11:39:21 GMT+00:00 schrieb Ben Cooksley <bcooksley at kde.org
> >:
>
>> Hi all,
>>
>> For some time now we have had a variety of issues with our Docker/Podman
>> based CI builds. These have included the lack of GUI test support on
>> Windows, periodic crashes on FreeBSD, poor IO performance of Windows
>> builds, issues supporting builds for Flatpak and Snaps and inability to
>> support either builds or tests where elevated privileges or system session
>> resources are needed.
>>
>> In addition to this we've had issues where Linux CI builds have the
>> capability to trigger OOM events on the CI hosts, which in turn takes out
>> Windows and (less often) FreeBSD builders. While this does not occur too
>> often, it does happen from time to time and eventually negatively impacts
>> the build queue for those platforms.
>>
>> The need to have dedicated VMs for FreeBSD and Windows on our builders
>> also makes setting up of a CI build node for KDE software a more
>> complicated and time intensive task than it otherwise needs to be (and
>> means that the amount of systems to care for increases by 3 for every CI
>> node we add).
>>
>> While individually relatively minor, together these issues more than
>> justify making a significant change to the way we run our CI system - in
>> this case transitioning from container based builds to VM based builds.
>>
>> These builds will still take place on dedicated hardware that we control,
>> however instead of taking place within a container (managed by Podman on
>> Linux and FreeBSD, or Docker on Windows) they will instead take place
>> within a VM using a copy-on-write disk image.
>> VM based builds will unfortunately take a little longer to start (it
>> takes ~10 seconds for a VM from any of Linux, FreeBSD or Windows to boot on
>> my personal system) however the benefits we gain should more than outweigh
>> this small downside.
>>
>> This has been under development for the past couple of weeks and is now
>> reaching the point where the only remaining steps are to get it integrated
>> with the Gitlab CI agent (gitlab-runner) for which prototype code is
>> already in place, and complete porting of our images over. Once that
>> happens a complete rebuild of all of our builders will be swiftly
>> undertaken to transition them completely over to the new VM based
>> infrastructure.
>>
>> Specs wise, at this time it is planned for each spawned standard VM to be
>> provided with 2/3's of the system CPU cores (so 12 cores), 16GB RAM and
>> 100GB of disk space (although some of that will be occupied by the system
>> image - approximately 10GB for standard Linux builds and ~30GB or so for
>> Windows builds). There will be a higher resource tier available for certain
>> builds however that will be on request only and would need to be justified
>> (such as Craft needing to build QtWebEngine).
>>
>> As launching VMs is not the most efficient approach for all workloads,
>> limited support for running Docker containers will be preserved, however
>> this support is primarily intended for running linters, sanity checks and
>> website builds, and is not intended for running general CI/CD builds.
>>
>> The tooling used by the CI nodes to run VMs is something that should be
>> fairly trivial for people to run on their own local system should they wish
>> to run any of those images (say for FreeBSD or Android), although you will
>> need to setup libvirt yourself (SUSE has very good instructions for this,
>> Debian less so as their instructions lack installing the packages needed to
>> provide UEFI and TPM support). The tooling itself was merged this evening
>> to sysadmin/ci-images (vm-common/ folder) and can be used with the VM
>> images found at https://storage.kde.org/vm-images/
>>
>> There is however one downside to this - Qt 5 support.
>>
>> Over the past few months distributions have been steadily removing
>> packages and other supporting infrastructure needed to keep Qt 5 builds
>> alive. In the case of Windows, support for the entire Qt 5 tree has been
>> unmaintained for some time. For FreeBSD and SUSE a significant number of
>> packages have been removed - which in the case of SUSE also includes
>> packages needed to support the building of KJS.  Accordingly, because
>> builds of Frameworks are a first stepping stone to support building
>> anything else, it will not be possible for us to produce Qt 5 based VM
>> build images for any of the 3 platforms.
>>
>> We will therefore have to remove Qt 5 support from the CI system with the
>> transition to VM based CI.
>>
>> Please let me know if there are any questions on the above.
>>
>> Thanks,
>> Ben
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.kde.org/pipermail/kde-devel/attachments/20250603/162316c1/attachment.htm>