Test Failures Fatal by Default in KDE CI

[Kristen McWilliam]

Hi all,

I'm excited to share an important update to our Continuous Integration
system that will improve the stability and reliability of our software.
Starting soon, test failures will be fatal by default in KDE CI.

Historically, making test failures fatal has been an opt-in feature. This
allowed some projects to inadvertently overlook failing tests, which could
lead to undetected breakages. To address this, we are transitioning to a
new default behavior where test failures will block merging, ensuring that
every commit keeps our code in a working state.


## Key Details

### New Opt-Out Option

We've introduced a new `allow-failing-tests-on` option for the
`.kde-ci.yml` file for projects that need temporary flexibility. If your
project isn't ready for this change, you can opt out using this option.

Previously, the `require-passing-tests-on` option was used to make test
failures fatal by adding it to the options in the `.kde-ci.yml` file like
this:

    require-passing-tests-on: [ 'Linux', 'Windows' ]

If you need to opt out of the new default behavior, you can add the
`allow-failing-tests-on` option to your `.kde-ci.yml` file like this:

    allow-failing-tests-on: [ 'Linux', 'Windows' ]

Full example of a `.kde-ci.yml` file with the new option:

```yaml
# SPDX-FileCopyrightText: None
# SPDX-License-Identifier: CC0-1.0

Dependencies:
- 'on': ['@all']
  'require':
    'frameworks/kcoreaddons': '@stable-kf6'

Options:
  allow-failing-tests-on: [ 'Windows' ]
```

The new opt-out option is available now, and the default behavior has **not
yet changed**. We will have a transition period to allow projects to adjust
before implementing the new default.


### Transition Period

There will be a 6-week transition period to allow projects to adjust before
making test failures fatal by default. During this time we will:

  - Monitor adoption and feedback.
  - Assist projects that haven't been adjusted yet, especially those under
community maintenance.


### Final Steps

After the transition period, test failures will become fatal by default.
Additionally:

  - The legacy `require-passing-tests-on` option will be removed.
  - We will communicate the change to the community via the mailing list
and Discourse.


### What You Need to Do

We encourage all maintainers to review and update their CI configurations
accordingly. Your proactive adjustments and feedback will help ensure a
smooth transition and maintain the high quality of KDE projects.

If your project needs to allow a platform to continue failing tests
temporarily, add the `allow-failing-tests-on` option to your `.kde-ci.yml`
file with the appropriate platforms listed.

If some tests are failing because the Windows CI can't handle certain GUI
tests, prefer conditionally skipping only those tests on Windows CI runs by
adding the following to the problematic tests:

```cpp
if (qEnvironmentVariableIntValue("KDECI_CANNOT_CREATE_WINDOWS")) {
    QSKIP("KDE CI can't create a window on this platform, skipping this
test");
}
```

If you have any questions or need support during this process, reply here
or in the accompanying Discourse thread:

https://discuss.kde.org/t/test-failures-fatal-by-default-in-kde-ci/31670


## TLDR

Test failures will be fatal by default in KDE CI. Use
`allow-failing-tests-on` to opt out temporarily. Review and update your CI
configurations to ensure a smooth transition. Reach out for support if
needed.

---

Thank you for your attention and commitment to maintaining robust,
high-quality code.

Cheers,
Kristen
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.kde.org/pipermail/kde-devel/attachments/20250317/bbe1149a/attachment.htm>