Plasma CI Health Check

[Justin Zobel]

This is great work! I love the dashboard. I've started doing a few repos 
changing them to:

Options:
require-passing-tests-on:['@all']

Regards,

Justin

On 11/5/26 22:42, Kristen McWilliam wrote:
> Hello, and welcome to the new Plasma CI Health Check!
>
> You may be familiar with the "KDE Gear projects with failing CI" 
> project/emails that Albert has been doing for some time — I will be 
> doing similar for Plasma, to ensure that our CI is healthy and improve 
> our testing strategy.
>
> We'll be looking at a few interesting metrics, for example whether:
>
> • CI is passing or failing on master
>
> If CI is failing on master its usually a bad sign, and something we 
> want to fix ASAP.
>
>
> • Tests are being automatically retried
>
> Having the CI configured to auto retry tests is usually a sign of a 
> workaround for flaky tests, and flaky tests are bad tests; at best 
> they are frustrating and a waste of CI resources, at worst they give a 
> false sense of security that code is actually reliable. A test should 
> either pass or fail — if it doesn't do so reliably it should be fixed 
> or removed.
>
> In the .kde-ci.yml this will look something like:
>
> ```yaml
> Options:
>  ctest-arguments: '--repeat until-pass:5'
> ```
>
> We should prefer to remove this option and fix the tests.
>
>
> • Tests are configured as required passing for the repo
>
> With this enabled, if tests fail in the CI the associated merge 
> request will show this as a nice obvious warning; if it isn't enabled, 
> the MR will report the pipeline is completely green — not good!
>
> We definitely want it to be obvious if the tests failed, otherwise 
> people are likely to merge it without even being aware there is an issue.
>
> In the .kde-ci.yml this will look like:
>
> ```yaml
> Options:
>  require-passing-tests-on: ['@all']
> ```
>
> Knowing if the tests are passing or failing is super important, so we 
> need to work towards this being enabled for all our repos.
>
>
> ---------------------------
>
>
> There will be additional things to track to assess our CI and test 
> health, but this is a good place to start.
>
> Here are our initial metrics:
>
> • 86 monitored repos
> • 10 failing CI in master
> • 3 configured for auto-repeating tests
> • 14% of repos configured to require passing tests
>
>
> Failing CI:
>
> • plasma/qqc2-breeze-style
> • plasma/plasma-desktop
> • plasma/kmenuedit
> • plasma/discover
> • plasma/lightdm-kde-greeter
> • plasma/kde-gtk-config
> • plasma/plasma-sdk
> • plasma/plasma-systemmonitor
> • plasma/latte-dock
> • plasma-mobile/spacebar
>
>
> Auto-retrying tests:
>
> • kwin-x11
> • kwin
> • plasma-desktop
>
>
> I hacked together a simple dashboard to help visualize the state of 
> things, which also includes links to the relevant CI run for each 
> repo; if interested in checking, it lives here for the moment:
>
> https://ci-healthcheck-5e5c80.local-kde.org/
>
>
> Thank you for helping to improve the reliability of our software, and 
> reducing wasted CI resources.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.kde.org/pipermail/plasma-devel/attachments/20260512/dcc1394d/attachment-0001.htm>