CI Utilisation and system efficiency

Sat Apr 19 09:47:13 BST 2025

On Sat, Apr 19, 2025 at 7:25 AM Ben Cooksley <bcooksley at kde.org> wrote:

> Hi all,
>
> Over the past week or two there have been a number of complaints regarding
> CI builder availability which i've done some investigating into this
> morning.
>
> Part of this is related to the Windows CI builders falling offline due to
> OOM events, however the rest is simply due to a lack of builder time
> availability (which is what this email is focused on).
>
> Given we have 6 Hetzner AX51 servers connected to Gitlab (each equipped
> with a Ryzen 7 7700 CPU, 64GB RAM and NVMe storage) the issue is not
> available build power - it is the number of builds and the length of those
> builds that is at issue.
>
> This morning I ran a basic query to ascertain the top 20 projects for CI
> time utilisation on invent.kde.org which revealed the following:
>
>           full_path           |    time_used     | job_count
> ------------------------------+------------------+-----------
> plasma/kwin                  | 320:47:04.966412 |      2387
> graphics/krita               | 178:03:19.080763 |       423
> multimedia/kdenlive          | 174:08:09.876842 |       697
> network/ruqola               | 173:17:47.311305 |       555
> plasma/plasma-workspace      | 155:10:03.618929 |       660
> network/neochat              | 138:03:23.926652 |      1546
> education/kstars             | 129:49:17.74229  |       329
> sysadmin/ci-management       | 111:21:09.739792 |       154
> plasma/plasma-desktop        | 108:56:52.849433 |       776
> kde-linux/kde-linux-packages | 81:00:10.001937  |        33
> kdevelop/kdevelop            | 59:40:51.54474   |       217
> office/kmymoney              | 54:32:00.24623   |       271
> frameworks/kio               | 53:54:19.046685  |       690
> education/labplot            | 52:36:30.343671  |       245
> murveit/kstars               | 52:32:56.882728  |       128
> frameworks/kirigami          | 47:07:19.172935  |      1627
> system/dolphin               | 46:09:58.02836   |       705
> kde-linux/kde-linux          | 39:25:54.052469  |        46
> utilities/kate               | 36:09:22.18958   |       356
> wreissenberger/kstars        | 35:58:14.120515  |       105
>
> If we look closely, KStars has three spots on this list (totalling 216
> hours of time used, making it the biggest app user of CI time).
>
> Projects on the above list are asked to please review their jobs and how
> they are conducting development to ensure CI time is used efficiently and
> appropriately.
>
> Other projects should also please review their usage and optimise
> accordingly even if they're not on this list as there is efficiencies to be
> found in all projects.
>
> When reviewing the list of CI builds projects have enabled, it is
> important to consider to what degree your project benefits from having
> various builds enabled. One common pattern i've seen is having Alpine, SUSE
> Qt 6.9 and SUSE Qt 6.10 all enabled.
>
> If you need to verify building on Alpine / MUSL type systems and wish to
> monitor for Qt Next regressions then you probably shouldn't have a
> conventional Linux Qt stable build as those two jobs between them already
> cover that list of permutations.
>

Thinking through this further...

All projects that have enabled Linux Qt 6.10 (Linux-Next) which also have
FreeBSD or Windows builds shouldn't really have Linux Qt 6.9 builds enabled
unless that job has other specific benefits (ie. Alpine and you're
monitoring buildability with another libc)

My logic behind this is:
- Linux Qt 6.10 monitors buildability on glibc + gcc + Qt 6.10
- Linux Qt 6.9 monitors buildability on glibc + gcc + Qt 6.9
- Alpine Qt 6.9 monitors buildability on musl + gcc + Qt 6.9
- FreeBSD Qt 6.9 monitors buildability on BSD libc + Clang + Qt 6.9
- Windows Qt 6.9 monitors buildability on MS CRT + MSVC + Qt 6.9

In the above combination sets, a project with Linux Qt 6.10 may also have
one or more of Alpine Qt 6.9, FreeBSD Qt 6.9 and Windows Qt 6.9.

It must never have Linux Qt 6.9 unless it is a Linux only project that does
not support musl - because coverage for glibc / gcc / Qt 6.9 has already
been achieved by Linux Qt 6.10 (for the first two) and any of the other
jobs (for Qt 6.9).

KWin and Ruqola must therefore under the above logic drop Linux Qt 6.9 as
they are using Linux Qt 6.10 and are making use of FreeBSD / Windows /
Alpine builds.

> I've taken a quick look at some of these and can suggest the following:
>
> KWin: it has two conventional Linux jobs (suse_qt69 and suse_qt610) plus a
> custom reduced feature set job. It seems like one of these conventional
> Linux jobs should be dropped.
>
> KStars: Appears to have a custom Linux job in addition to a conventional
> Linux job. Choose one please.
>
> Ruqola: Appears to be conducting a development process whereby changes are
> made in stable then immediately merged to master in a ever continuing loop.
> Please discontinue this behaviour and only periodically merge stable to
> master.
>
> Also needs to drop one of it's Linux jobs as they're duplicating
> functionality as noted above.
>
> Plasma Workspace/Desktop: At least in part this seems to be driven by
> Appium tests. Please reduce the number of these and/or streamline the
> process for running an Appium test. Consideration should be given to
> enabling the CI option use-ccache as well.
>
> KDevelop: Please enable the CI option use-ccache.
>
> Labplot: Appears to have a strange customisation in place to the standard
> jobs which shouldn't be necessary as flags in .kde-ci.yml should permit
> that to be done.
>
> Thanks,
> Ben
>

Cheers,
Ben
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.kde.org/pipermail/kdenlive/attachments/20250419/eff1c7e0/attachment-0001.htm>