<div dir="ltr"><div dir="ltr">On Wed, Mar 26, 2025 at 9:33 PM Albert Astals Cid <<a href="mailto:aacid@kde.org">aacid@kde.org</a>> wrote:</div><div class="gmail_quote gmail_quote_container"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">El dimecres, 26 de març del 2025, a les 8:38:38 (Hora estàndard d’Europa <br>
central), Ingo Klöcker va escriure:<br>
> On Dienstag, 25. März 2025 22:14:26 Mitteleuropäische Normalzeit Albert<br>
> Astals<br>
> Cid wrote:<br>
> > Lots Of Craft Projects - NEW<br>
> > <br>
> >  * There are too many craft jobs failed, timed-out or still waiting to<br>
> >  list.<br>
> > <br>
> > * My understanding is that all the notary services are down and causing<br>
> > those timeouts.<br>
> <br>
> I very much appreciate your work, but it would be helpful if you'd include<br>
> at least one link to such a job.<br>
<br>
<a href="https://invent.kde.org/utilities/kate/-/pipelines/916610" rel="noreferrer" target="_blank">https://invent.kde.org/utilities/kate/-/pipelines/916610</a>        <br>
<a href="https://invent.kde.org/system/dolphin/-/pipelines/917091" rel="noreferrer" target="_blank">https://invent.kde.org/system/dolphin/-/pipelines/917091</a><br>
<a href="https://invent.kde.org/education/kbruch/-/pipelines/917206" rel="noreferrer" target="_blank">https://invent.kde.org/education/kbruch/-/pipelines/917206</a><br>
<a href="https://invent.kde.org/education/kgeography/-/pipelines/917208" rel="noreferrer" target="_blank">https://invent.kde.org/education/kgeography/-/pipelines/917208</a><br>
<a href="https://invent.kde.org/education/kiten/-/pipelines/917210" rel="noreferrer" target="_blank">https://invent.kde.org/education/kiten/-/pipelines/917210</a><br>
<a href="https://invent.kde.org/education/kturtle/-/pipelines/917215" rel="noreferrer" target="_blank">https://invent.kde.org/education/kturtle/-/pipelines/917215</a><br>
<a href="https://invent.kde.org/education/kwordquiz/-/pipelines/916415" rel="noreferrer" target="_blank">https://invent.kde.org/education/kwordquiz/-/pipelines/916415</a><br>
<a href="https://invent.kde.org/education/parley/-/pipelines/917219" rel="noreferrer" target="_blank">https://invent.kde.org/education/parley/-/pipelines/917219</a><br>
<a href="https://invent.kde.org/graphics/okular/-/pipelines/918126" rel="noreferrer" target="_blank">https://invent.kde.org/graphics/okular/-/pipelines/918126</a><br>
<a href="https://invent.kde.org/utilities/ark/-/pipelines/917946" rel="noreferrer" target="_blank">https://invent.kde.org/utilities/ark/-/pipelines/917946</a><br>
<a href="https://invent.kde.org/utilities/filelight/-/pipelines/917646" rel="noreferrer" target="_blank">https://invent.kde.org/utilities/filelight/-/pipelines/917646</a><br>
<a href="https://invent.kde.org/games/bomber/-/pipelines/917249" rel="noreferrer" target="_blank">https://invent.kde.org/games/bomber/-/pipelines/917249</a><br>
<a href="https://invent.kde.org/games/bovo/-/pipelines/917250" rel="noreferrer" target="_blank">https://invent.kde.org/games/bovo/-/pipelines/917250</a><br>
<a href="https://invent.kde.org/games/granatier/-/pipelines/917251" rel="noreferrer" target="_blank">https://invent.kde.org/games/granatier/-/pipelines/917251</a><br>
<a href="https://invent.kde.org/games/kapman/-/pipelines/917253" rel="noreferrer" target="_blank">https://invent.kde.org/games/kapman/-/pipelines/917253</a><br>
<a href="https://invent.kde.org/games/katomic/-/pipelines/917254" rel="noreferrer" target="_blank">https://invent.kde.org/games/katomic/-/pipelines/917254</a><br>
<a href="https://invent.kde.org/games/kblackbox/-/pipelines/917255" rel="noreferrer" target="_blank">https://invent.kde.org/games/kblackbox/-/pipelines/917255</a><br>
<a href="https://invent.kde.org/games/kblocks/-/pipelines/917256" rel="noreferrer" target="_blank">https://invent.kde.org/games/kblocks/-/pipelines/917256</a><br>
<a href="https://invent.kde.org/games/kbounce/-/pipelines/917257" rel="noreferrer" target="_blank">https://invent.kde.org/games/kbounce/-/pipelines/917257</a><br>
<a href="https://invent.kde.org/games/kbreakout/-/pipelines/917258" rel="noreferrer" target="_blank">https://invent.kde.org/games/kbreakout/-/pipelines/917258</a><br>
<a href="https://invent.kde.org/games/kdiamond/-/pipelines/917259" rel="noreferrer" target="_blank">https://invent.kde.org/games/kdiamond/-/pipelines/917259</a><br>
<a href="https://invent.kde.org/games/kfourinline/-/pipelines/917260" rel="noreferrer" target="_blank">https://invent.kde.org/games/kfourinline/-/pipelines/917260</a><br>
<a href="https://invent.kde.org/games/kgoldrunner/-/pipelines/917261" rel="noreferrer" target="_blank">https://invent.kde.org/games/kgoldrunner/-/pipelines/917261</a><br>
<a href="https://invent.kde.org/games/kigo/-/pipelines/917262" rel="noreferrer" target="_blank">https://invent.kde.org/games/kigo/-/pipelines/917262</a><br>
<a href="https://invent.kde.org/games/killbots/-/pipelines/917263" rel="noreferrer" target="_blank">https://invent.kde.org/games/killbots/-/pipelines/917263</a><br>
<a href="https://invent.kde.org/games/kiriki/-/pipelines/917264" rel="noreferrer" target="_blank">https://invent.kde.org/games/kiriki/-/pipelines/917264</a><br>
<a href="https://invent.kde.org/games/kjumpingcube/-/pipelines/917265" rel="noreferrer" target="_blank">https://invent.kde.org/games/kjumpingcube/-/pipelines/917265</a><br>
<a href="https://invent.kde.org/games/klickety/-/pipelines/917266" rel="noreferrer" target="_blank">https://invent.kde.org/games/klickety/-/pipelines/917266</a><br>
<a href="https://invent.kde.org/games/klines/-/pipelines/917267" rel="noreferrer" target="_blank">https://invent.kde.org/games/klines/-/pipelines/917267</a><br>
<a href="https://invent.kde.org/games/kmahjongg/-/pipelines/917268" rel="noreferrer" target="_blank">https://invent.kde.org/games/kmahjongg/-/pipelines/917268</a><br>
<a href="https://invent.kde.org/games/kmines/-/pipelines/917269" rel="noreferrer" target="_blank">https://invent.kde.org/games/kmines/-/pipelines/917269</a><br>
<a href="https://invent.kde.org/games/knetwalk/-/pipelines/917271" rel="noreferrer" target="_blank">https://invent.kde.org/games/knetwalk/-/pipelines/917271</a><br>
<a href="https://invent.kde.org/games/kolf/-/pipelines/917273" rel="noreferrer" target="_blank">https://invent.kde.org/games/kolf/-/pipelines/917273</a><br>
<a href="https://invent.kde.org/games/kollision/-/pipelines/917274" rel="noreferrer" target="_blank">https://invent.kde.org/games/kollision/-/pipelines/917274</a><br>
<a href="https://invent.kde.org/games/konquest/-/pipelines/917275" rel="noreferrer" target="_blank">https://invent.kde.org/games/konquest/-/pipelines/917275</a><br>
<a href="https://invent.kde.org/games/kreversi/-/pipelines/917277" rel="noreferrer" target="_blank">https://invent.kde.org/games/kreversi/-/pipelines/917277</a><br>
<a href="https://invent.kde.org/games/kshisen/-/pipelines/917278" rel="noreferrer" target="_blank">https://invent.kde.org/games/kshisen/-/pipelines/917278</a><br>
<a href="https://invent.kde.org/games/ksnakeduel/-/pipelines/917280" rel="noreferrer" target="_blank">https://invent.kde.org/games/ksnakeduel/-/pipelines/917280</a><br>
<a href="https://invent.kde.org/games/kspaceduel/-/pipelines/917281" rel="noreferrer" target="_blank">https://invent.kde.org/games/kspaceduel/-/pipelines/917281</a><br>
<a href="https://invent.kde.org/games/ksquares/-/pipelines/917282" rel="noreferrer" target="_blank">https://invent.kde.org/games/ksquares/-/pipelines/917282</a><br>
<a href="https://invent.kde.org/games/ksudoku/-/pipelines/916599" rel="noreferrer" target="_blank">https://invent.kde.org/games/ksudoku/-/pipelines/916599</a><br>
<a href="https://invent.kde.org/games/ktuberling/-/pipelines/917283" rel="noreferrer" target="_blank">https://invent.kde.org/games/ktuberling/-/pipelines/917283</a><br>
<a href="https://invent.kde.org/games/kubrick/-/pipelines/917284" rel="noreferrer" target="_blank">https://invent.kde.org/games/kubrick/-/pipelines/917284</a><br>
<a href="https://invent.kde.org/games/lskat/-/pipelines/917287" rel="noreferrer" target="_blank">https://invent.kde.org/games/lskat/-/pipelines/917287</a><br>
<a href="https://invent.kde.org/games/palapeli/-/pipelines/917288" rel="noreferrer" target="_blank">https://invent.kde.org/games/palapeli/-/pipelines/917288</a><br>
<a href="https://invent.kde.org/games/picmi/-/pipelines/917289" rel="noreferrer" target="_blank">https://invent.kde.org/games/picmi/-/pipelines/917289</a><br>
<a href="https://invent.kde.org/sdk/lokalize/-/pipelines/916311" rel="noreferrer" target="_blank">https://invent.kde.org/sdk/lokalize/-/pipelines/916311</a><br>
<a href="https://invent.kde.org/pim/kontact/-/pipelines/918216" rel="noreferrer" target="_blank">https://invent.kde.org/pim/kontact/-/pipelines/918216</a><br>
<a href="https://invent.kde.org/pim/pim-sieve-editor/-/pipelines/917322" rel="noreferrer" target="_blank">https://invent.kde.org/pim/pim-sieve-editor/-/pipelines/917322</a><br>
<a href="https://invent.kde.org/multimedia/elisa/-/pipelines/917333" rel="noreferrer" target="_blank">https://invent.kde.org/multimedia/elisa/-/pipelines/917333</a><br>
<a href="https://invent.kde.org/network/kdeconnect-kde/-/pipelines/917334" rel="noreferrer" target="_blank">https://invent.kde.org/network/kdeconnect-kde/-/pipelines/917334</a><br>
<a href="https://invent.kde.org/network/konversation/-/pipelines/917340" rel="noreferrer" target="_blank">https://invent.kde.org/network/konversation/-/pipelines/917340</a><br>
<a href="https://invent.kde.org/kdevelop/kdevelop/-/pipelines/917345" rel="noreferrer" target="_blank">https://invent.kde.org/kdevelop/kdevelop/-/pipelines/917345</a><br>
<a href="https://invent.kde.org/multimedia/kasts/-/pipelines/916345" rel="noreferrer" target="_blank">https://invent.kde.org/multimedia/kasts/-/pipelines/916345</a><br>
<a href="https://invent.kde.org/network/neochat/-/pipelines/917915" rel="noreferrer" target="_blank">https://invent.kde.org/network/neochat/-/pipelines/917915</a></blockquote><div><br></div><div>There are a variety of issues driving those failures.</div><div><br></div><div>On the Windows failures, it looks like the Windows 11 system that is used to run the signer got itself internally confused and was therefore no longer able to perform signing operations.</div><div>The only cure for this ironically is a reboot, which I have now completed.</div><div><br></div><div>On the build publisher failures, that was due to a hung rsync process which was blocking the currently running job at the time. The rsync process hung because the remote side ran out of disk space. The remote side issue was fixed not long after it occurred however it wasn't noticed until now that the build publisher notary was stuck. This led to all jobs involving the build publisher getting stuck since that happened. I've now unstuck the build publisher and have returned it to normal operation.</div><div><br></div><div>The signers especially for Windows are facing elevated traffic whenever a mingw build runs as well because there is a issue with building Attica apparently.</div><div>See <a href="https://invent.kde.org/sysadmin/ci-management/-/jobs/2738569">https://invent.kde.org/sysadmin/ci-management/-/jobs/2738569</a> for the build log. That is preventing a cache update from completing and means that every build of a MingW project (such as Kdenlive) is sending a signing request for each dependency it compiles. That will be aggravating the Windows signer backlog and will have also contributed to the failures / timeouts you've seen.</div><div><br></div><div>On the MacOS front, it looks like one of the machines has fallen asleep, so there is currently only one builder which won't have helped matters there. Just in case the Mac notary services were experiencing after effects from the lack of disk space, I have also restarted those services however they appear to be functioning normally now as far as I can tell.</div><div><br></div><div><br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
<br>
<br>
<br>
<br>
<br>
> A few minutes ago I checked the Windows<br>
> notary (triggered by your message on Matrix which I saw just before going<br>
> to bed yesterday) and it was up and running. Maybe Ben restarted it<br>
> already.<br>
> <br>
> The Android notary services are all up and running since the last restart 17<br>
> days ago after config changes (and they have been up and running flawlessly<br>
> for months).<br>
> <br>
> The only notary service that regularly goes offline is the Windows signer<br>
> because Microsoft regularly reboots the Windows VM after installing updates<br>
> (and we have not yet set up the Windows signer to start automatically after<br>
> a reboot).<br>
> <br>
> Regards,<br>
> Ingo<br></blockquote><div><br></div><div>Cheers,</div><div>Ben </div></div></div>