Upcoming CI changes - transition to VM based CI

[Ben Cooksley]

On Mon, Jun 9, 2025 at 8:10 PM Albert Astals Cid <aacid at kde.org> wrote:

> El dimecres, 4 de juny del 2025, a les 14:27:35 (Hora d’estiu d’Europa
> central), Ben Cooksley va escriure:
> > On Wed, Jun 4, 2025 at 8:17 AM Albert Astals Cid <aacid at kde.org> wrote:
> > > El dimarts, 3 de juny del 2025, a les 11:42:08 (Hora d’estiu d’Europa
> > >
> > > central), Ben Cooksley va escriure:
> > > > On Tue, Jun 3, 2025 at 9:03 AM Albert Astals Cid <aacid at kde.org>
> wrote:
> > > > > El dilluns, 2 de juny del 2025, a les 13:39:21 (Hora d’estiu
> d’Europa
> > > > >
> > > > > central), Ben Cooksley va escriure:
> > > > > > Hi all,
> > > > > >
> > > > > > For some time now we have had a variety of issues with our
> > >
> > > Docker/Podman
> > >
> > > > > > based CI builds. These have included the lack of GUI test
> support on
> > > > > > Windows, periodic crashes on FreeBSD, poor IO performance of
> Windows
> > > > > > builds, issues supporting builds for Flatpak and Snaps and
> inability
> > >
> > > to
> > >
> > > > > > support either builds or tests where elevated privileges or
> system
> > > > >
> > > > > session
> > > > >
> > > > > > resources are needed.
> > > > > >
> > > > > > In addition to this we've had issues where Linux CI builds have
> the
> > > > > > capability to trigger OOM events on the CI hosts, which in turn
> > > > > > takes
> > > > > > out
> > > > > > Windows and (less often) FreeBSD builders. While this does not
> occur
> > >
> > > too
> > >
> > > > > > often, it does happen from time to time and eventually negatively
> > > > > > impacts
> > > > > > the build queue for those platforms.
> > > > > >
> > > > > > The need to have dedicated VMs for FreeBSD and Windows on our
> > >
> > > builders
> > >
> > > > > also
> > > > >
> > > > > > makes setting up of a CI build node for KDE software a more
> > >
> > > complicated
> > >
> > > > > and
> > > > >
> > > > > > time intensive task than it otherwise needs to be (and means that
> > > > > > the
> > > > > > amount of systems to care for increases by 3 for every CI node we
> > >
> > > add).
> > >
> > > > > > While individually relatively minor, together these issues more
> than
> > > > > > justify making a significant change to the way we run our CI
> system
> > >
> > > - in
> > >
> > > > > > this case transitioning from container based builds to VM based
> > >
> > > builds.
> > >
> > > > > > These builds will still take place on dedicated hardware that we
> > > > > > control,
> > > > > > however instead of taking place within a container (managed by
> > >
> > > Podman on
> > >
> > > > > > Linux and FreeBSD, or Docker on Windows) they will instead take
> > > > > > place
> > > > > > within a VM using a copy-on-write disk image.
> > > > > > VM based builds will unfortunately take a little longer to start
> (it
> > > > >
> > > > > takes
> > > > >
> > > > > > ~10 seconds for a VM from any of Linux, FreeBSD or Windows to
> boot
> > >
> > > on my
> > >
> > > > > > personal system) however the benefits we gain should more than
> > >
> > > outweigh
> > >
> > > > > > this small downside.
> > > > > >
> > > > > > This has been under development for the past couple of weeks and
> is
> > >
> > > now
> > >
> > > > > > reaching the point where the only remaining steps are to get it
> > > > >
> > > > > integrated
> > > > >
> > > > > > with the Gitlab CI agent (gitlab-runner) for which prototype
> code is
> > > > > > already in place, and complete porting of our images over. Once
> that
> > > > > > happens a complete rebuild of all of our builders will be swiftly
> > > > > > undertaken to transition them completely over to the new VM based
> > > > > > infrastructure.
> > > > > >
> > > > > > Specs wise, at this time it is planned for each spawned standard
> VM
> > >
> > > to
> > >
> > > > > > be
> > > > > > provided with 2/3's of the system CPU cores (so 12 cores), 16GB
> RAM
> > >
> > > and
> > >
> > > > > > 100GB of disk space (although some of that will be occupied by
> the
> > > > > > system
> > > > > > image - approximately 10GB for standard Linux builds and ~30GB
> or so
> > >
> > > for
> > >
> > > > > > Windows builds). There will be a higher resource tier available
> for
> > > > >
> > > > > certain
> > > > >
> > > > > > builds however that will be on request only and would need to be
> > > > >
> > > > > justified
> > > > >
> > > > > > (such as Craft needing to build QtWebEngine).
> > > > > >
> > > > > > As launching VMs is not the most efficient approach for all
> > >
> > > workloads,
> > >
> > > > > > limited support for running Docker containers will be preserved,
> > >
> > > however
> > >
> > > > > > this support is primarily intended for running linters, sanity
> > > > > > checks
> > > > > > and
> > > > > > website builds, and is not intended for running general CI/CD
> > > > > > builds.
> > > > > >
> > > > > > The tooling used by the CI nodes to run VMs is something that
> should
> > >
> > > be
> > >
> > > > > > fairly trivial for people to run on their own local system should
> > >
> > > they
> > >
> > > > > wish
> > > > >
> > > > > > to run any of those images (say for FreeBSD or Android), although
> > > > > > you
> > > > >
> > > > > will
> > > > >
> > > > > > need to setup libvirt yourself (SUSE has very good instructions
> for
> > > > > > this,
> > > > > > Debian less so as their instructions lack installing the packages
> > >
> > > needed
> > >
> > > > > to
> > > > >
> > > > > > provide UEFI and TPM support). The tooling itself was merged this
> > > > > > evening
> > > > > > to sysadmin/ci-images (vm-common/ folder) and can be used with
> the
> > > > > > VM
> > > > > > images found at https://storage.kde.org/vm-images/
> > > > > >
> > > > > > There is however one downside to this - Qt 5 support.
> > > > > >
> > > > > > Over the past few months distributions have been steadily
> removing
> > > > >
> > > > > packages
> > > > >
> > > > > > and other supporting infrastructure needed to keep Qt 5 builds
> > >
> > > alive. In
> > >
> > > > > > the case of Windows, support for the entire Qt 5 tree has been
> > > > >
> > > > > unmaintained
> > > > >
> > > > > > for some time. For FreeBSD and SUSE a significant number of
> packages
> > > > > > have
> > > > > > been removed - which in the case of SUSE also includes packages
> > >
> > > needed
> > >
> > > > > > to
> > > > > > support the building of KJS.  Accordingly, because builds of
> > >
> > > Frameworks
> > >
> > > > > are
> > > > >
> > > > > > a first stepping stone to support building anything else, it will
> > >
> > > not be
> > >
> > > > > > possible for us to produce Qt 5 based VM build images for any of
> the
> > >
> > > 3
> > >
> > > > > > platforms.
> > > > > >
> > > > > > We will therefore have to remove Qt 5 support from the CI system
> > > > > > with
> > > > > > the
> > > > > > transition to VM based CI.
> > > > >
> > > > > From previous discussions I had the impression this was only for
> > > > > things
> > > > > that
> > > > > wanted to create packages and not for "want to have CI to
> compile/run
> > > > > tests".
> > > > >
> > > > > Can you confirm you are proposng a total annihilation of Qt5
> support
> > > > > in
> > > > > our
> > > > > CI?
> > > >
> > > > At the time we had that discussion it was still possible to build
> some
> > > > of
> > > > the Qt 5 images, however that is no longer the case - all of them now
> > >
> > > fail
> > >
> > > > to build.
> > > >
> > > > In the case of the suse-qt515 image, the removal of libpcre in SUSE
> > > > means
> > > > it is no longer possible to build KJS.
> > > > Consequently, we're no longer able to build all Frameworks (making
> 'kf5'
> > > > branch CI for Frameworks non-functional) so there isn't much point in
> > > > looking further to support Qt 5 on Linux.
> > > >
> > > > For FreeBSD the story is much the same as SUSE - packages are being
> > >
> > > removed
> > >
> > > > as apps upgrade to Qt 6 and the Qt 5 version of libraries becomes
> > > > surplus
> > > > to requirements.
> > > >
> > > > For Windows the continued operation of that CI has only been possible
> > > > because our existing images are still around - new ones cannot be
> built.
> > > > That has been the case for a significant amount of time now, and it
> is
> > >
> > > not
> > >
> > > > worth the investment to fix it as everyone who works on Windows has
> > > > moved
> > > > on to Qt 6.
> > > >
> > > > In essence there is little we can do to keep this alive -
> distributions
> > >
> > > are
> > >
> > > > removing support so we must follow suit.
> > > >
> > > > The correct course of action is to accelerate the porting of the
> > >
> > > remaining
> > >
> > > > applications, not to delay and keep Qt 5 alive.
> > >
> > > As Nico said, we can just not build KJS/KHTML and then give the
> projects
> > > that
> > > still use Qt5 a reasonable timeframe for them to port to Qt6, e.g.
> until
> > > the
> > > end of the year, not until the end of the month.
> >
> > Depending on how projects have written their .kde-ci.yml files, this will
> > either work fine or fail so a degree of rewriting of those files may be
> > needed.
> > But it could be tried - it is very much keeping something on true last
> gasp
> > life support though and isn't going to be pleasant to maintain as things
> > are going to keep disappearing.
>
> If this is too hard can we just suggest these projects to use flatpak as a
> CI?
>
> We already have a "modern" SDK built there so folks could be using that
> and I
> guess it would not give us much problems in regards to how CI is setup?
>

Flatpak would not provide unit test support i'm afraid.

I did manage (after stripping a series of packages out) to get the image to
build successfully.
It remains to be seen whether it is able to actually perform a build, but
that will be tested soon enough.


>
> >
> > My concern is that if we keep pushing out terminating support those
> > projects will just keep deferring their porting.
> > Note that Frameworks 6 released in Feb 2024, so they've had 15 months so
> > far to port which is a pretty reasonable amount of time.
>
> As something to compare to, we kept shipping kdelibs4 based apps in KDE
> Gear
> until April 2018 when KDE Frameworks 5.0 was released in July 2014.
>
> That's almost 4 years, I am not asking for that, it's not that 15 months
> is a
> reasonable time or not, it's that we never set a line on the sand and thus
> it's easy to keep postponing it because it's not "there yet". So I would
> propose setting a date in the sand when we will stop supporting Qt5 stuff
> in
> our CI system. My proposal would be the end of 2025.
>

I'd like it to be a bit earlier than that if we can - say 30 September.
Quite a few applications have made it off since the initial effort to start
pulling the plug and another bunch have been pulled out in response to the
recent effort to remove Qt 5 CI support.


>
> >
> > Can we at least go through and remove Qt 5 CI from anything that has
> ported
> > over and is running both currently?
> > Given we're now unable to build them we should probably declare KHTML and
> > KJS unmaintained and archive them as well.
>
> I guess we can live with that.
>
> Cheers,
>   Albert
>

Thanks,
Ben


>
> >
> > > Cheers,
> > >
> > >   Albert
> >
> > Regards,
> > Ben
> >
> > > > > Cheers,
> > > > >
> > > > >   Albert
> > > >
> > > > Regards,
> > > > Ben
> > > >
> > > > > > Please let me know if there are any questions on the above.
> > > > > >
> > > > > > Thanks,
> > > > > > Ben
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.kde.org/pipermail/kde-core-devel/attachments/20250610/cc287c51/attachment-0001.htm>