Plasma CI Health Check

[Kristen McWilliam]

Excellent, thank you for doing so! :)

On 5/11/26 10:00 PM, Justin Zobel wrote:
>
> This is great work! I love the dashboard. I've started doing a few 
> repos changing them to:
>
> Options:
> require-passing-tests-on:['@all']
>
> Regards,
>
> Justin
>
> On 11/5/26 22:42, Kristen McWilliam wrote:
>> Hello, and welcome to the new Plasma CI Health Check!
>>
>> You may be familiar with the "KDE Gear projects with failing CI" 
>> project/emails that Albert has been doing for some time — I will be 
>> doing similar for Plasma, to ensure that our CI is healthy and 
>> improve our testing strategy.
>>
>> We'll be looking at a few interesting metrics, for example whether:
>>
>> • CI is passing or failing on master
>>
>> If CI is failing on master its usually a bad sign, and something we 
>> want to fix ASAP.
>>
>>
>> • Tests are being automatically retried
>>
>> Having the CI configured to auto retry tests is usually a sign of a 
>> workaround for flaky tests, and flaky tests are bad tests; at best 
>> they are frustrating and a waste of CI resources, at worst they give 
>> a false sense of security that code is actually reliable. A test 
>> should either pass or fail — if it doesn't do so reliably it should 
>> be fixed or removed.
>>
>> In the .kde-ci.yml this will look something like:
>>
>> ```yaml
>> Options:
>>  ctest-arguments: '--repeat until-pass:5'
>> ```
>>
>> We should prefer to remove this option and fix the tests.
>>
>>
>> • Tests are configured as required passing for the repo
>>
>> With this enabled, if tests fail in the CI the associated merge 
>> request will show this as a nice obvious warning; if it isn't 
>> enabled, the MR will report the pipeline is completely green — not good!
>>
>> We definitely want it to be obvious if the tests failed, otherwise 
>> people are likely to merge it without even being aware there is an 
>> issue.
>>
>> In the .kde-ci.yml this will look like:
>>
>> ```yaml
>> Options:
>>  require-passing-tests-on: ['@all']
>> ```
>>
>> Knowing if the tests are passing or failing is super important, so we 
>> need to work towards this being enabled for all our repos.
>>
>>
>> ---------------------------
>>
>>
>> There will be additional things to track to assess our CI and test 
>> health, but this is a good place to start.
>>
>> Here are our initial metrics:
>>
>> • 86 monitored repos
>> • 10 failing CI in master
>> • 3 configured for auto-repeating tests
>> • 14% of repos configured to require passing tests
>>
>>
>> Failing CI:
>>
>> • plasma/qqc2-breeze-style
>> • plasma/plasma-desktop
>> • plasma/kmenuedit
>> • plasma/discover
>> • plasma/lightdm-kde-greeter
>> • plasma/kde-gtk-config
>> • plasma/plasma-sdk
>> • plasma/plasma-systemmonitor
>> • plasma/latte-dock
>> • plasma-mobile/spacebar
>>
>>
>> Auto-retrying tests:
>>
>> • kwin-x11
>> • kwin
>> • plasma-desktop
>>
>>
>> I hacked together a simple dashboard to help visualize the state of 
>> things, which also includes links to the relevant CI run for each 
>> repo; if interested in checking, it lives here for the moment:
>>
>> https://ci-healthcheck-5e5c80.local-kde.org/
>>
>>
>> Thank you for helping to improve the reliability of our software, and 
>> reducing wasted CI resources.
>>
-- 
Cheers,
   Kristen
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.kde.org/pipermail/plasma-devel/attachments/20260512/1cbafb1f/attachment.htm>