Test Failures Fatal by Default in KDE CI

[Albert Astals Cid]

El dilluns, 17 de març del 2025, a les 14:54:24 (Hora estàndard d’Europa 
central), Kristen McWilliam va escriure:
> Hi all,
> 
> I'm excited to share an important update to our Continuous Integration
> system that will improve the stability and reliability of our software.
> Starting soon, test failures will be fatal by default in KDE CI.
> 
> Historically, making test failures fatal has been an opt-in feature. This
> allowed some projects to inadvertently overlook failing tests, which could
> lead to undetected breakages. To address this, we are transitioning to a
> new default behavior where test failures will block merging, ensuring that
> every commit keeps our code in a working state.
> 
> 
> ## Key Details
> 
> ### New Opt-Out Option
> 
> We've introduced a new `allow-failing-tests-on` option for the
> `.kde-ci.yml` file for projects that need temporary flexibility. If your
> project isn't ready for this change, you can opt out using this option.
> 
> Previously, the `require-passing-tests-on` option was used to make test
> failures fatal by adding it to the options in the `.kde-ci.yml` file like
> this:
> 
>     require-passing-tests-on: [ 'Linux', 'Windows' ]
> 
> If you need to opt out of the new default behavior, you can add the
> `allow-failing-tests-on` option to your `.kde-ci.yml` file like this:
> 
>     allow-failing-tests-on: [ 'Linux', 'Windows' ]
> 
> Full example of a `.kde-ci.yml` file with the new option:
> 
> ```yaml
> # SPDX-FileCopyrightText: None
> # SPDX-License-Identifier: CC0-1.0
> 
> Dependencies:
> - 'on': ['@all']
>   'require':
>     'frameworks/kcoreaddons': '@stable-kf6'
> 
> Options:
>   allow-failing-tests-on: [ 'Windows' ]
> ```
> 
> The new opt-out option is available now, and the default behavior has **not
> yet changed**. We will have a transition period to allow projects to adjust
> before implementing the new default.
> 
> 
> ### Transition Period
> 
> There will be a 6-week transition period to allow projects to adjust before
> making test failures fatal by default. During this time we will:
> 
>   - Monitor adoption and feedback.
>   - Assist projects that haven't been adjusted yet, especially those under
> community maintenance.
> 
> 
> ### Final Steps
> 
> After the transition period, test failures will become fatal by default.
> Additionally:
> 
>   - The legacy `require-passing-tests-on` option will be removed.
>   - We will communicate the change to the community via the mailing list
> and Discourse.
> 
> 
> ### What You Need to Do
> 
> We encourage all maintainers to review and update their CI configurations
> accordingly. Your proactive adjustments and feedback will help ensure a
> smooth transition and maintain the high quality of KDE projects.
> 
> If your project needs to allow a platform to continue failing tests
> temporarily, add the `allow-failing-tests-on` option to your `.kde-ci.yml`
> file with the appropriate platforms listed.
> 
> If some tests are failing because the Windows CI can't handle certain GUI
> tests, prefer conditionally skipping only those tests on Windows CI runs by
> adding the following to the problematic tests:
> 
> ```cpp
> if (qEnvironmentVariableIntValue("KDECI_CANNOT_CREATE_WINDOWS")) {
>     QSKIP("KDE CI can't create a window on this platform, skipping this
> test");
> }
> ```
> 
> If you have any questions or need support during this process, reply here
> or in the accompanying Discourse thread:
> 
> https://discuss.kde.org/t/test-failures-fatal-by-default-in-kde-ci/31670
> 
> 
> ## TLDR
> 
> Test failures will be fatal by default in KDE CI. Use
> `allow-failing-tests-on` to opt out temporarily. Review and update your CI
> configurations to ensure a smooth transition. Reach out for support if
> needed.

Two questions:

A) Where was this discussed? I am relatively involved in getting people to get 
tests passing, etc. and had no idea this was being worked on, if i had known 
it would have saved me creating all the MRs i created recently for KDE Gear to 
enable forcing tests to pass.

B) Who is going to update all the repositories in KDE Gear (or elsewhere but 
KDE Gear is my particular worry) that are community maintained (i.e. don't 
have a clear maintainer)?

Cheers,
  Albert

> 
> ---
> 
> Thank you for your attention and commitment to maintaining robust,
> high-quality code.
> 
> Cheers,
> Kristen