CI Utilisation and system efficiency

[Ben Cooksley]

On Sat, May 3, 2025 at 8:26 AM Christoph Cullmann <christoph at cullmann.io>
wrote:

>
>
> On Friday, May 2nd, 2025 at 22:18, Ben Cooksley <bcooksley at kde.org> wrote:
>
> On Sat, May 3, 2025 at 3:27 AM Christoph Cullmann <christoph at cullmann.io>
> wrote:
>
>>
>> Hi,
>>
>> at work we use cmake unity build to save time & costs.
>>
>> Would that be some idea here, too?
>>
>> Naturally as side effect that can hide compile issues
>> or introduce ones.
>>
>
> We could look into that, just not sure how it would save compile times
> though?
> We have extremely fast NVMe storage on the builders, so we're unlikely to
> be IO constrained.
>
>
> Hi,
>
> that saves more compute power than IO, you parse headers just 1/x of the
> time for x sized units
> and you spawn just 1/x of the processes, at work, that leads to 2 or 4
> times the speed, on bare metal
> machines with only SSDs, too.
>
> But naturally that will depend on the projects.
>

I see. Based on the CMake documentation it sounds like something that
doesn't always work so i'm hesitant to enable it as a global flag.
The CI system through the option cmake-options in .kde-ci.yml can allow for
the necessary unity build flags to be passed - so this is something heavy
users with lots of files may want to look into.


>
> Greetings
> Christoph
>

Cheers,
Ben


>
>
>
>> Greetings
>> Christoph
>>
>
> Thanks,
> Ben
>
>>
>> On Friday, April 18th, 2025 at 21:25, Ben Cooksley <bcooksley at kde.org>
>> wrote:
>>
>> Hi all,
>>
>> Over the past week or two there have been a number of complaints
>> regarding CI builder availability which i've done some investigating into
>> this morning.
>>
>> Part of this is related to the Windows CI builders falling offline due to
>> OOM events, however the rest is simply due to a lack of builder time
>> availability (which is what this email is focused on).
>>
>> Given we have 6 Hetzner AX51 servers connected to Gitlab (each equipped
>> with a Ryzen 7 7700 CPU, 64GB RAM and NVMe storage) the issue is not
>> available build power - it is the number of builds and the length of those
>> builds that is at issue.
>>
>> This morning I ran a basic query to ascertain the top 20 projects for CI
>> time utilisation on invent.kde.org which revealed the following:
>>
>> full_path | time_used | job_count
>> ------------------------------+------------------+-----------
>> plasma/kwin | 320:47:04.966412 | 2387
>> graphics/krita | 178:03:19.080763 | 423
>> multimedia/kdenlive | 174:08:09.876842 | 697
>> network/ruqola | 173:17:47.311305 | 555
>> plasma/plasma-workspace | 155:10:03.618929 | 660
>> network/neochat | 138:03:23.926652 | 1546
>> education/kstars | 129:49:17.74229 | 329
>> sysadmin/ci-management | 111:21:09.739792 | 154
>> plasma/plasma-desktop | 108:56:52.849433 | 776
>> kde-linux/kde-linux-packages | 81:00:10.001937 | 33
>> kdevelop/kdevelop | 59:40:51.54474 | 217
>> office/kmymoney | 54:32:00.24623 | 271
>> frameworks/kio | 53:54:19.046685 | 690
>> education/labplot | 52:36:30.343671 | 245
>> murveit/kstars | 52:32:56.882728 | 128
>> frameworks/kirigami | 47:07:19.172935 | 1627
>> system/dolphin | 46:09:58.02836 | 705
>> kde-linux/kde-linux | 39:25:54.052469 | 46
>> utilities/kate | 36:09:22.18958 | 356
>> wreissenberger/kstars | 35:58:14.120515 | 105
>>
>> If we look closely, KStars has three spots on this list (totalling 216
>> hours of time used, making it the biggest app user of CI time).
>>
>> Projects on the above list are asked to please review their jobs and how
>> they are conducting development to ensure CI time is used efficiently and
>> appropriately.
>>
>> Other projects should also please review their usage and optimise
>> accordingly even if they're not on this list as there is efficiencies to be
>> found in all projects.
>>
>> When reviewing the list of CI builds projects have enabled, it is
>> important to consider to what degree your project benefits from having
>> various builds enabled. One common pattern i've seen is having Alpine, SUSE
>> Qt 6.9 and SUSE Qt 6.10 all enabled.
>>
>> If you need to verify building on Alpine / MUSL type systems and wish to
>> monitor for Qt Next regressions then you probably shouldn't have a
>> conventional Linux Qt stable build as those two jobs between them already
>> cover that list of permutations.
>>
>> I've taken a quick look at some of these and can suggest the following:
>>
>> KWin: it has two conventional Linux jobs (suse_qt69 and suse_qt610) plus
>> a custom reduced feature set job. It seems like one of these conventional
>> Linux jobs should be dropped.
>>
>> KStars: Appears to have a custom Linux job in addition to a conventional
>> Linux job. Choose one please.
>>
>> Ruqola: Appears to be conducting a development process whereby changes
>> are made in stable then immediately merged to master in a ever continuing
>> loop. Please discontinue this behaviour and only periodically merge stable
>> to master.
>>
>> Also needs to drop one of it's Linux jobs as they're duplicating
>> functionality as noted above.
>>
>> Plasma Workspace/Desktop: At least in part this seems to be driven by
>> Appium tests. Please reduce the number of these and/or streamline the
>> process for running an Appium test. Consideration should be given to
>> enabling the CI option use-ccache as well.
>>
>> KDevelop: Please enable the CI option use-ccache.
>>
>> Labplot: Appears to have a strange customisation in place to the standard
>> jobs which shouldn't be necessary as flags in .kde-ci.yml should permit
>> that to be done.
>>
>> Thanks,
>> Ben
>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.kde.org/pipermail/kde-devel/attachments/20250503/bcecd55c/attachment-0001.htm>