CI Utilisation and system efficiency

Tue May 6 16:43:45 BST 2025

On Dienstag, 6. Mai 2025 11:26:30 Mitteleuropäische Sommerzeit Ben Cooksley 
wrote:
> On Tue, May 6, 2025 at 5:36 AM Volker Krause <vkrause at kde.org> wrote:
> > On Freitag, 2. Mai 2025 22:55:34 Mitteleuropäische Sommerzeit Ben Cooksley
> > 
> > wrote:
> > > On Sat, May 3, 2025 at 8:26 AM Christoph Cullmann <christoph at cullmann.io
> > > 
> > > wrote:
> > > > On Friday, May 2nd, 2025 at 22:18, Ben Cooksley <bcooksley at kde.org>
> > 
> > wrote:
> > > > On Sat, May 3, 2025 at 3:27 AM Christoph Cullmann <
> > 
> > christoph at cullmann.io>
> > 
> > > > wrote:
> > > >> Hi,
> > > >> 
> > > >> at work we use cmake unity build to save time & costs.
> > > >> 
> > > >> Would that be some idea here, too?
> > > >> 
> > > >> Naturally as side effect that can hide compile issues
> > > >> or introduce ones.
> > > > 
> > > > We could look into that, just not sure how it would save compile times
> > > > though?
> > > > We have extremely fast NVMe storage on the builders, so we're unlikely
> > 
> > to
> > 
> > > > be IO constrained.
> > > > 
> > > > 
> > > > Hi,
> > > > 
> > > > that saves more compute power than IO, you parse headers just 1/x of
> > 
> > the
> > 
> > > > time for x sized units
> > > > and you spawn just 1/x of the processes, at work, that leads to 2 or 4
> > > > times the speed, on bare metal
> > > > machines with only SSDs, too.
> > > > 
> > > > But naturally that will depend on the projects.
> > > 
> > > I see. Based on the CMake documentation it sounds like something that
> > > doesn't always work so i'm hesitant to enable it as a global flag.
> > > The CI system through the option cmake-options in .kde-ci.yml can allow
> > 
> > for
> > 
> > > the necessary unity build flags to be passed - so this is something
> > > heavy
> > > users with lots of files may want to look into.
> > 
> > Right, this is probably not a suitable default.
> > 
> > What might be worth exploring though is using unity builds for QML
> > cachegen
> > code by default. That's many somewhat uniform and predictable files with
> > the
> > same includes etc. And at least for several things in the top 20 those
> > files
> > are a significant part of the cost.
> 
> This sounds like something that needs to be sorted within ECM / Frameworks
> / the applications themselves though and isn't something that can be done
> at a global CI level?

Right. Here's a quick prototype implementing this in ECM:
https://invent.kde.org/frameworks/extra-cmake-modules/-/merge_requests/525

Saves about 12% job time and 15% CPU time on a clean Kirigami build. Obviously 
something you'd pay back with slower incremental builds. If this is robust 
enough using this as a CI-only optimization might give us the best of both 
worlds though.

Another thing I noticed that is indeed more on the CI side to fix: Seed jobs 
build unit tests currently.

Regards,
Volker

> > > >> On Friday, April 18th, 2025 at 21:25, Ben Cooksley <bcooksley at kde.org
> > > >> 
> > > >> wrote:
> > > >> 
> > > >> Hi all,
> > > >> 
> > > >> Over the past week or two there have been a number of complaints
> > > >> regarding CI builder availability which i've done some investigating
> > 
> > into
> > 
> > > >> this morning.
> > > >> 
> > > >> Part of this is related to the Windows CI builders falling offline
> > 
> > due to
> > 
> > > >> OOM events, however the rest is simply due to a lack of builder time
> > > >> availability (which is what this email is focused on).
> > > >> 
> > > >> Given we have 6 Hetzner AX51 servers connected to Gitlab (each
> > 
> > equipped
> > 
> > > >> with a Ryzen 7 7700 CPU, 64GB RAM and NVMe storage) the issue is not
> > > >> available build power - it is the number of builds and the length of
> > > >> those
> > > >> builds that is at issue.
> > > >> 
> > > >> This morning I ran a basic query to ascertain the top 20 projects for
> > 
> > CI
> > 
> > > >> time utilisation on invent.kde.org which revealed the following:
> > > >> 
> > > >> full_path | time_used | job_count
> > > >> ------------------------------+------------------+-----------
> > > >> plasma/kwin | 320:47:04.966412 | 2387
> > > >> graphics/krita | 178:03:19.080763 | 423
> > > >> multimedia/kdenlive | 174:08:09.876842 | 697
> > > >> network/ruqola | 173:17:47.311305 | 555
> > > >> plasma/plasma-workspace | 155:10:03.618929 | 660
> > > >> network/neochat | 138:03:23.926652 | 1546
> > > >> education/kstars | 129:49:17.74229 | 329
> > > >> sysadmin/ci-management | 111:21:09.739792 | 154
> > > >> plasma/plasma-desktop | 108:56:52.849433 | 776
> > > >> kde-linux/kde-linux-packages | 81:00:10.001937 | 33
> > > >> kdevelop/kdevelop | 59:40:51.54474 | 217
> > > >> office/kmymoney | 54:32:00.24623 | 271
> > > >> frameworks/kio | 53:54:19.046685 | 690
> > > >> education/labplot | 52:36:30.343671 | 245
> > > >> murveit/kstars | 52:32:56.882728 | 128
> > > >> frameworks/kirigami | 47:07:19.172935 | 1627
> > > >> system/dolphin | 46:09:58.02836 | 705
> > > >> kde-linux/kde-linux | 39:25:54.052469 | 46
> > > >> utilities/kate | 36:09:22.18958 | 356
> > > >> wreissenberger/kstars | 35:58:14.120515 | 105
> > > >> 
> > > >> If we look closely, KStars has three spots on this list (totalling
> > > >> 216
> > > >> hours of time used, making it the biggest app user of CI time).
> > > >> 
> > > >> Projects on the above list are asked to please review their jobs and
> > 
> > how
> > 
> > > >> they are conducting development to ensure CI time is used efficiently
> > 
> > and
> > 
> > > >> appropriately.
> > > >> 
> > > >> Other projects should also please review their usage and optimise
> > > >> accordingly even if they're not on this list as there is efficiencies
> > 
> > to
> > 
> > > >> be
> > > >> found in all projects.
> > > >> 
> > > >> When reviewing the list of CI builds projects have enabled, it is
> > > >> important to consider to what degree your project benefits from
> > > >> having
> > > >> various builds enabled. One common pattern i've seen is having
> > > >> Alpine,
> > > >> SUSE
> > > >> Qt 6.9 and SUSE Qt 6.10 all enabled.
> > > >> 
> > > >> If you need to verify building on Alpine / MUSL type systems and wish
> > 
> > to
> > 
> > > >> monitor for Qt Next regressions then you probably shouldn't have a
> > > >> conventional Linux Qt stable build as those two jobs between them
> > 
> > already
> > 
> > > >> cover that list of permutations.
> > > >> 
> > > >> I've taken a quick look at some of these and can suggest the
> > 
> > following:
> > > >> KWin: it has two conventional Linux jobs (suse_qt69 and suse_qt610)
> > 
> > plus
> > 
> > > >> a custom reduced feature set job. It seems like one of these
> > 
> > conventional
> > 
> > > >> Linux jobs should be dropped.
> > > >> 
> > > >> KStars: Appears to have a custom Linux job in addition to a
> > 
> > conventional
> > 
> > > >> Linux job. Choose one please.
> > > >> 
> > > >> Ruqola: Appears to be conducting a development process whereby
> > > >> changes
> > > >> are made in stable then immediately merged to master in a ever
> > 
> > continuing
> > 
> > > >> loop. Please discontinue this behaviour and only periodically merge
> > > >> stable
> > > >> to master.
> > > >> 
> > > >> Also needs to drop one of it's Linux jobs as they're duplicating
> > > >> functionality as noted above.
> > > >> 
> > > >> Plasma Workspace/Desktop: At least in part this seems to be driven by
> > > >> Appium tests. Please reduce the number of these and/or streamline the
> > > >> process for running an Appium test. Consideration should be given to
> > > >> enabling the CI option use-ccache as well.
> > > >> 
> > > >> KDevelop: Please enable the CI option use-ccache.
> > > >> 
> > > >> Labplot: Appears to have a strange customisation in place to the
> > 
> > standard
> > 
> > > >> jobs which shouldn't be necessary as flags in .kde-ci.yml should
> > 
> > permit
> > 
> > > >> that to be done.
> > > >> 
> > > >> Thanks,
> > > >> Ben

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 195 bytes
Desc: This is a digitally signed message part.
URL: <http://mail.kde.org/pipermail/kde-devel/attachments/20250506/fe0836c9/attachment.sig>