Plasma CI Health Check

[Justin Zobel]

On 11 May 2026 10:42:39 pm ACST, Kristen McWilliam <kristen at kde.org> wrote:
>Hello, and welcome to the new Plasma CI Health Check!
>
>You may be familiar with the "KDE Gear projects with failing CI" project/emails that Albert has been doing for some time — I will be doing similar for Plasma, to ensure that our CI is healthy and improve our testing strategy.
>
>We'll be looking at a few interesting metrics, for example whether:
>
>• CI is passing or failing on master
>
>If CI is failing on master its usually a bad sign, and something we want to fix ASAP.
>
>
>• Tests are being automatically retried
>
>Having the CI configured to auto retry tests is usually a sign of a workaround for flaky tests, and flaky tests are bad tests; at best they are frustrating and a waste of CI resources, at worst they give a false sense of security that code is actually reliable. A test should either pass or fail — if it doesn't do so reliably it should be fixed or removed.
>
>In the .kde-ci.yml this will look something like:
>
>```yaml
>Options:
> ctest-arguments: '--repeat until-pass:5'
>```
>
>We should prefer to remove this option and fix the tests.
>
>
>• Tests are configured as required passing for the repo
>
>With this enabled, if tests fail in the CI the associated merge request will show this as a nice obvious warning; if it isn't enabled, the MR will report the pipeline is completely green — not good!
>
>We definitely want it to be obvious if the tests failed, otherwise people are likely to merge it without even being aware there is an issue.
>
>In the .kde-ci.yml this will look like:
>
>```yaml
>Options:
> require-passing-tests-on: ['@all']
>```
>
>Knowing if the tests are passing or failing is super important, so we need to work towards this being enabled for all our repos.
>
>
>---------------------------
>
>
>There will be additional things to track to assess our CI and test health, but this is a good place to start.
>
>Here are our initial metrics:
>
>• 86 monitored repos
>• 10 failing CI in master
>• 3 configured for auto-repeating tests
>• 14% of repos configured to require passing tests
>
>
>Failing CI:
>
>• plasma/qqc2-breeze-style
>• plasma/plasma-desktop
>• plasma/kmenuedit
>• plasma/discover
>• plasma/lightdm-kde-greeter
>• plasma/kde-gtk-config
>• plasma/plasma-sdk
>• plasma/plasma-systemmonitor
>• plasma/latte-dock
>• plasma-mobile/spacebar
>
>
>Auto-retrying tests:
>
>• kwin-x11
>• kwin
>• plasma-desktop
>
>
>I hacked together a simple dashboard to help visualize the state of things, which also includes links to the relevant CI run for each repo; if interested in checking, it lives here for the moment:
>
>https://ci-healthcheck-5e5c80.local-kde.org/
>
>
>Thank you for helping to improve the reliability of our software, and reducing wasted CI resources.
>
>-- 
>Cheers,
>  Kristen
>

If that is your idea of a simple dashboard, mine are straight up Neanderthal. Great work!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.kde.org/pipermail/plasma-devel/attachments/20260512/22a049a6/attachment.htm>