CI congestion/starvation
Ben Cooksley
bcooksley at kde.org
Sun Feb 15 18:26:17 GMT 2026
On Mon, Feb 16, 2026 at 3:52 AM Alexander Semke <alexander.semke at web.de>
wrote:
> On 13/02/26 12:37, Ben Cooksley wrote:
> > [...]
> > Resource utilisation wise, i've not looked into whether there has been
> > a significant bump in the number of jobs, but over the past year some
> > additional CD support has been added so that indicates some trouble
> there.
> > System utilisation stats since the start of February are rather telling
> >
> > full_path | time_used | job_count
> > ------------------------------+------------------+-----------
> > graphics/krita | 228:03:25.300251 | 451
> > plasma/kwin | 164:34:16.119352 | 2032
> > plasma/plasma-workspace | 136:03:27.534157 | 1044
> > multimedia/kdenlive | 113:13:39.087112 | 566
> > network/ruqola | 89:02:24.704107 | 369
> > pim/messagelib | 83:58:38.936333 | 648
> > graphics/drawy | 77:38:55.987716 | 2101
> > network/neochat | 76:37:12.290371 | 1069
> > plasma/plasma-desktop | 62:27:04.468738 | 619
> > utilities/kate | 59:05:46.517227 | 438
> > office/kmymoney | 56:58:08.349153 | 241
> > education/labplot | 55:19:06.879103 | 423
> > frameworks/kio | 43:04:55.33379 | 744
> > kde-linux/kde-linux-packages | 39:40:15.332536 | 63
> > libraries/ktextaddons | 31:46:16.168865 | 356
> > office/calligra | 31:33:35.74806 | 62
> > kdevelop/kdevelop | 30:57:38.655126 | 113
> > education/kstars | 30:04:42.328143 | 71
> > sysadmin/craft-ci | 28:36:13.927477 | 53
> > bjordan/kdenlive | 24:13:20.564568 | 191
> >
> >
> > Contrast that with the full month of September 2025:
> >
> > full_path | time_used | job_count
> > -------------------------------+------------------+-----------
> > graphics/krita | 382:38:14.237481 | 882
> > plasma/kwin | 365:27:01.96681 | 3531
> > multimedia/kdenlive | 294:59:01.874773 | 1299
> > network/neochat | 233:26:31.601019 | 2298
> > packaging/flatpak-kde-runtime | 225:01:38.542109 | 269
> > network/ruqola | 219:56:34.415103 | 946
> > plasma/plasma-workspace | 194:15:09.699541 | 2014
> > sysadmin/ci-management | 118:10:28.754041 | 233
> > libraries/ktextaddons | 105:25:58.724964 | 1084
> > office/kmymoney | 101:50:27.612055 | 572
> > plasma/plasma-desktop | 101:25:05.185953 | 1148
> > kde-linux/kde-linux-packages | 101:17:06.284095 | 144
> > education/rkward | 84:32:07.118316 | 1033
> > kdevelop/kdevelop | 84:04:58.804139 | 270
> > utilities/kate | 83:49:15.561603 | 663
> > frameworks/kio | 76:46:04.377597 | 1032
> > graphics/okular | 74:58:19.8886 | 628
> > pim/akonadi | 62:23:56.252831 | 531
> > pim/itinerary | 60:05:52.269941 | 578
> > network/kaidan | 56:37:44.446695 | 1773
>
>
> Thanks Ben, this is an interesting information. Would it be feasible for
> you to send such a report to the mailing list on the monthly base? Or
> even add this as the output for every merge request on gitlab? I think
> teams can probably easily improve in the various areas, at least in some
> of them, and optimize the build and the tests to reduce the overall
> runtime. Providing more transparency into the consumption of resources
> on CI per project, per job and per test can support such optimizations
> and will also help to raise the awareness for the impact of certain
> changes on CI.
>
It has to be manually put together based on queries from the database, so
can't be put on every MR.
It can be run periodically though.
>
> For me, waiting for CI is really annoying and also disrupting personal
> workflows sometimes (waiting today for 3h already in
> https://invent.kde.org/education/labplot/-/merge_requests/845, gave up
> and switched to a different branch already and will need to go back in
> case of errors). For LabPlot we did already multiple optimizations in
Looks like we dropped 2 runners again (node2 and node3) which i've now
revived.
This hasn't been helped by a number of large projects doing quite a bit of
activity today (which is unusual for a Sunday)
full_path | time_used | job_count
------------------------------------+-----------------+-----------
plasma/plasma-desktop | 07:25:58.851394 | 44
graphics/krita | 07:16:57.78504 | 12
multimedia/kdenlive | 05:17:33.323612 | 30
office/kmymoney | 03:54:42.976902 | 16
pehg/kaidan | 03:16:03.91073 | 30
snap-packaging/kf6-core-sdk | 03:15:04.291499 | 3
system/dolphin | 03:10:27.969557 | 48
utilities/ark | 03:04:04.142 | 33
network/neochat | 02:59:31.366648 | 85
education/labplot | 02:50:22.827995 | 23
libraries/ktextaddons | 02:44:16.520082 | 35
pim/messagelib | 02:17:45.340389 | 14
pim/itinerary | 02:14:12.165157 | 8
cwo/plasma-desktop | 02:09:26.453516 | 7
deeptangshusaha/kdeconnect-android | 01:42:42.818736 | 9
office/skrooge | 01:35:08.920711 | 9
qazcetelic/kdeconnect-android | 01:13:10.125977 | 6
graphics/drawy | 01:11:45.894211 | 44
lito/kdeconnect-kde | 01:06:17.659945 | 24
graphics/digikam | 01:05:11.594924 | 13
>
> the past - not because of the long pending time but just because the
> runtime itself was painful and we plan to do another iteration and to
> invest more here soon. The problem with the pending time come on top now
> and I think teams should check what can be improved here in general
> without putting the focus on more hardware only.
>
Optimisations, especially when they are things that should be done anyway -
like removing tests that are broken (timeout, fail) or which don't actually
test anything of meaning - are always useful.
I do believe in this instance there is a hardware element to it - even if
it is to simply improve reliability.
i've noticed that crashes tend to happen most frequently during the launch
stage as well, so i'm considering adding locking to vm-runner so only one
VM can be starting at any given moment - there is a possibility we are
triggering a bug in KVM itself (or elsewhere in the kernel as shipped by
Debian). Given the reliability has been declining since the initial
introduction of VM based CI i'm not 100% convinced this is the case though.
>
>
> --
>
> Alexander
>
>
Cheers,
Ben
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.kde.org/pipermail/kde-devel/attachments/20260216/759fb4cc/attachment-0001.htm>
More information about the kde-devel
mailing list