Plasma CI Health Check

[Kristen McWilliam]

Hello, and welcome to the new Plasma CI Health Check!

You may be familiar with the "KDE Gear projects with failing CI" 
project/emails that Albert has been doing for some time — I will be 
doing similar for Plasma, to ensure that our CI is healthy and improve 
our testing strategy.

We'll be looking at a few interesting metrics, for example whether:

• CI is passing or failing on master

If CI is failing on master its usually a bad sign, and something we want 
to fix ASAP.


• Tests are being automatically retried

Having the CI configured to auto retry tests is usually a sign of a 
workaround for flaky tests, and flaky tests are bad tests; at best they 
are frustrating and a waste of CI resources, at worst they give a false 
sense of security that code is actually reliable. A test should either 
pass or fail — if it doesn't do so reliably it should be fixed or removed.

In the .kde-ci.yml this will look something like:

```yaml
Options:
  ctest-arguments: '--repeat until-pass:5'
```

We should prefer to remove this option and fix the tests.


• Tests are configured as required passing for the repo

With this enabled, if tests fail in the CI the associated merge request 
will show this as a nice obvious warning; if it isn't enabled, the MR 
will report the pipeline is completely green — not good!

We definitely want it to be obvious if the tests failed, otherwise 
people are likely to merge it without even being aware there is an issue.

In the .kde-ci.yml this will look like:

```yaml
Options:
  require-passing-tests-on: ['@all']
```

Knowing if the tests are passing or failing is super important, so we 
need to work towards this being enabled for all our repos.


---------------------------


There will be additional things to track to assess our CI and test 
health, but this is a good place to start.

Here are our initial metrics:

• 86 monitored repos
• 10 failing CI in master
• 3 configured for auto-repeating tests
• 14% of repos configured to require passing tests


Failing CI:

• plasma/qqc2-breeze-style
• plasma/plasma-desktop
• plasma/kmenuedit
• plasma/discover
• plasma/lightdm-kde-greeter
• plasma/kde-gtk-config
• plasma/plasma-sdk
• plasma/plasma-systemmonitor
• plasma/latte-dock
• plasma-mobile/spacebar


Auto-retrying tests:

• kwin-x11
• kwin
• plasma-desktop


I hacked together a simple dashboard to help visualize the state of 
things, which also includes links to the relevant CI run for each repo; 
if interested in checking, it lives here for the moment:

https://ci-healthcheck-5e5c80.local-kde.org/


Thank you for helping to improve the reliability of our software, and 
reducing wasted CI resources.

-- 
Cheers,
   Kristen