CI Utilisation and system efficiency

[Ben Cooksley]

Hi all,

Over the past week or two there have been a number of complaints regarding
CI builder availability which i've done some investigating into this
morning.

Part of this is related to the Windows CI builders falling offline due to
OOM events, however the rest is simply due to a lack of builder time
availability (which is what this email is focused on).

Given we have 6 Hetzner AX51 servers connected to Gitlab (each equipped
with a Ryzen 7 7700 CPU, 64GB RAM and NVMe storage) the issue is not
available build power - it is the number of builds and the length of those
builds that is at issue.

This morning I ran a basic query to ascertain the top 20 projects for CI
time utilisation on invent.kde.org which revealed the following:

          full_path           |    time_used     | job_count
------------------------------+------------------+-----------
plasma/kwin                  | 320:47:04.966412 |      2387
graphics/krita               | 178:03:19.080763 |       423
multimedia/kdenlive          | 174:08:09.876842 |       697
network/ruqola               | 173:17:47.311305 |       555
plasma/plasma-workspace      | 155:10:03.618929 |       660
network/neochat              | 138:03:23.926652 |      1546
education/kstars             | 129:49:17.74229  |       329
sysadmin/ci-management       | 111:21:09.739792 |       154
plasma/plasma-desktop        | 108:56:52.849433 |       776
kde-linux/kde-linux-packages | 81:00:10.001937  |        33
kdevelop/kdevelop            | 59:40:51.54474   |       217
office/kmymoney              | 54:32:00.24623   |       271
frameworks/kio               | 53:54:19.046685  |       690
education/labplot            | 52:36:30.343671  |       245
murveit/kstars               | 52:32:56.882728  |       128
frameworks/kirigami          | 47:07:19.172935  |      1627
system/dolphin               | 46:09:58.02836   |       705
kde-linux/kde-linux          | 39:25:54.052469  |        46
utilities/kate               | 36:09:22.18958   |       356
wreissenberger/kstars        | 35:58:14.120515  |       105

If we look closely, KStars has three spots on this list (totalling 216
hours of time used, making it the biggest app user of CI time).

Projects on the above list are asked to please review their jobs and how
they are conducting development to ensure CI time is used efficiently and
appropriately.

Other projects should also please review their usage and optimise
accordingly even if they're not on this list as there is efficiencies to be
found in all projects.

When reviewing the list of CI builds projects have enabled, it is important
to consider to what degree your project benefits from having various builds
enabled. One common pattern i've seen is having Alpine, SUSE Qt 6.9 and
SUSE Qt 6.10 all enabled.

If you need to verify building on Alpine / MUSL type systems and wish to
monitor for Qt Next regressions then you probably shouldn't have a
conventional Linux Qt stable build as those two jobs between them already
cover that list of permutations.

I've taken a quick look at some of these and can suggest the following:

KWin: it has two conventional Linux jobs (suse_qt69 and suse_qt610) plus a
custom reduced feature set job. It seems like one of these conventional
Linux jobs should be dropped.

KStars: Appears to have a custom Linux job in addition to a conventional
Linux job. Choose one please.

Ruqola: Appears to be conducting a development process whereby changes are
made in stable then immediately merged to master in a ever continuing loop.
Please discontinue this behaviour and only periodically merge stable to
master.

Also needs to drop one of it's Linux jobs as they're duplicating
functionality as noted above.

Plasma Workspace/Desktop: At least in part this seems to be driven by
Appium tests. Please reduce the number of these and/or streamline the
process for running an Appium test. Consideration should be given to
enabling the CI option use-ccache as well.

KDevelop: Please enable the CI option use-ccache.

Labplot: Appears to have a strange customisation in place to the standard
jobs which shouldn't be necessary as flags in .kde-ci.yml should permit
that to be done.

Thanks,
Ben
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.kde.org/pipermail/kdenlive/attachments/20250419/e4415df8/attachment.htm>