KDE Gear projects with failing CI (master) (25 March 2025)

[Ben Cooksley]

On Wed, Mar 26, 2025 at 9:33 PM Albert Astals Cid <aacid at kde.org> wrote:

> El dimecres, 26 de març del 2025, a les 8:38:38 (Hora estàndard d’Europa
> central), Ingo Klöcker va escriure:
> > On Dienstag, 25. März 2025 22:14:26 Mitteleuropäische Normalzeit Albert
> > Astals
> > Cid wrote:
> > > Lots Of Craft Projects - NEW
> > >
> > >  * There are too many craft jobs failed, timed-out or still waiting to
> > >  list.
> > >
> > > * My understanding is that all the notary services are down and causing
> > > those timeouts.
> >
> > I very much appreciate your work, but it would be helpful if you'd
> include
> > at least one link to such a job.
>
> https://invent.kde.org/utilities/kate/-/pipelines/916610
> https://invent.kde.org/system/dolphin/-/pipelines/917091
> https://invent.kde.org/education/kbruch/-/pipelines/917206
> https://invent.kde.org/education/kgeography/-/pipelines/917208
> https://invent.kde.org/education/kiten/-/pipelines/917210
> https://invent.kde.org/education/kturtle/-/pipelines/917215
> https://invent.kde.org/education/kwordquiz/-/pipelines/916415
> https://invent.kde.org/education/parley/-/pipelines/917219
> https://invent.kde.org/graphics/okular/-/pipelines/918126
> https://invent.kde.org/utilities/ark/-/pipelines/917946
> https://invent.kde.org/utilities/filelight/-/pipelines/917646
> https://invent.kde.org/games/bomber/-/pipelines/917249
> https://invent.kde.org/games/bovo/-/pipelines/917250
> https://invent.kde.org/games/granatier/-/pipelines/917251
> https://invent.kde.org/games/kapman/-/pipelines/917253
> https://invent.kde.org/games/katomic/-/pipelines/917254
> https://invent.kde.org/games/kblackbox/-/pipelines/917255
> https://invent.kde.org/games/kblocks/-/pipelines/917256
> https://invent.kde.org/games/kbounce/-/pipelines/917257
> https://invent.kde.org/games/kbreakout/-/pipelines/917258
> https://invent.kde.org/games/kdiamond/-/pipelines/917259
> https://invent.kde.org/games/kfourinline/-/pipelines/917260
> https://invent.kde.org/games/kgoldrunner/-/pipelines/917261
> https://invent.kde.org/games/kigo/-/pipelines/917262
> https://invent.kde.org/games/killbots/-/pipelines/917263
> https://invent.kde.org/games/kiriki/-/pipelines/917264
> https://invent.kde.org/games/kjumpingcube/-/pipelines/917265
> https://invent.kde.org/games/klickety/-/pipelines/917266
> https://invent.kde.org/games/klines/-/pipelines/917267
> https://invent.kde.org/games/kmahjongg/-/pipelines/917268
> https://invent.kde.org/games/kmines/-/pipelines/917269
> https://invent.kde.org/games/knetwalk/-/pipelines/917271
> https://invent.kde.org/games/kolf/-/pipelines/917273
> https://invent.kde.org/games/kollision/-/pipelines/917274
> https://invent.kde.org/games/konquest/-/pipelines/917275
> https://invent.kde.org/games/kreversi/-/pipelines/917277
> https://invent.kde.org/games/kshisen/-/pipelines/917278
> https://invent.kde.org/games/ksnakeduel/-/pipelines/917280
> https://invent.kde.org/games/kspaceduel/-/pipelines/917281
> https://invent.kde.org/games/ksquares/-/pipelines/917282
> https://invent.kde.org/games/ksudoku/-/pipelines/916599
> https://invent.kde.org/games/ktuberling/-/pipelines/917283
> https://invent.kde.org/games/kubrick/-/pipelines/917284
> https://invent.kde.org/games/lskat/-/pipelines/917287
> https://invent.kde.org/games/palapeli/-/pipelines/917288
> https://invent.kde.org/games/picmi/-/pipelines/917289
> https://invent.kde.org/sdk/lokalize/-/pipelines/916311
> https://invent.kde.org/pim/kontact/-/pipelines/918216
> https://invent.kde.org/pim/pim-sieve-editor/-/pipelines/917322
> https://invent.kde.org/multimedia/elisa/-/pipelines/917333
> https://invent.kde.org/network/kdeconnect-kde/-/pipelines/917334
> https://invent.kde.org/network/konversation/-/pipelines/917340
> https://invent.kde.org/kdevelop/kdevelop/-/pipelines/917345
> https://invent.kde.org/multimedia/kasts/-/pipelines/916345
> https://invent.kde.org/network/neochat/-/pipelines/917915


There are a variety of issues driving those failures.

On the Windows failures, it looks like the Windows 11 system that is used
to run the signer got itself internally confused and was therefore no
longer able to perform signing operations.
The only cure for this ironically is a reboot, which I have now completed.

On the build publisher failures, that was due to a hung rsync process which
was blocking the currently running job at the time. The rsync process hung
because the remote side ran out of disk space. The remote side issue was
fixed not long after it occurred however it wasn't noticed until now that
the build publisher notary was stuck. This led to all jobs involving the
build publisher getting stuck since that happened. I've now unstuck the
build publisher and have returned it to normal operation.

The signers especially for Windows are facing elevated traffic whenever a
mingw build runs as well because there is a issue with building Attica
apparently.
See https://invent.kde.org/sysadmin/ci-management/-/jobs/2738569 for the
build log. That is preventing a cache update from completing and means that
every build of a MingW project (such as Kdenlive) is sending a signing
request for each dependency it compiles. That will be aggravating the
Windows signer backlog and will have also contributed to the failures /
timeouts you've seen.

On the MacOS front, it looks like one of the machines has fallen asleep, so
there is currently only one builder which won't have helped matters there.
Just in case the Mac notary services were experiencing after effects from
the lack of disk space, I have also restarted those services however they
appear to be functioning normally now as far as I can tell.




>
>
>
>
>
> > A few minutes ago I checked the Windows
> > notary (triggered by your message on Matrix which I saw just before going
> > to bed yesterday) and it was up and running. Maybe Ben restarted it
> > already.
> >
> > The Android notary services are all up and running since the last
> restart 17
> > days ago after config changes (and they have been up and running
> flawlessly
> > for months).
> >
> > The only notary service that regularly goes offline is the Windows signer
> > because Microsoft regularly reboots the Windows VM after installing
> updates
> > (and we have not yet set up the Windows signer to start automatically
> after
> > a reboot).
> >
> > Regards,
> > Ingo
>

Cheers,
Ben
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.kde.org/pipermail/kde-devel/attachments/20250326/c3aabada/attachment.htm>