Test Failures Fatal by Default in KDE CI

[Albert Astals Cid]

El dimecres, 19 de març del 2025, a les 16:22:40 (Hora estàndard d’Europa 
central), Kristen McWilliam va escriure:
> A) Discussion was initially in
> https://invent.kde.org/sysadmin/ci-utilities/-/merge_requests/296 and then
> moved to https://invent.kde.org/sysadmin/ci-utilities/-/issues/37 for
> overall tracking. Sorry to hear you did that extra work; we're hoping to
> consolidate the effort with this. :)
> 
> B) No one has made any specific commitments to helping with this, but I
> know I for one will be trying my best to help where needed, such as the
> community maintained repos mentioned. Any assistance with that is obviously
> very welcome.

Thanks :)

Cheers,
  Albert

P.S: Please make sure you also answer to the list :)

> 
> 
> Cheers,
> Kristen
> 
> On Wed, Mar 19, 2025 at 10:48 AM Albert Astals Cid <aacid at kde.org> wrote:
> > El dilluns, 17 de març del 2025, a les 14:54:24 (Hora estàndard d’Europa
> > 
> > central), Kristen McWilliam va escriure:
> > > Hi all,
> > > 
> > > I'm excited to share an important update to our Continuous Integration
> > > system that will improve the stability and reliability of our software.
> > > Starting soon, test failures will be fatal by default in KDE CI.
> > > 
> > > Historically, making test failures fatal has been an opt-in feature.
> > > This
> > > allowed some projects to inadvertently overlook failing tests, which
> > 
> > could
> > 
> > > lead to undetected breakages. To address this, we are transitioning to a
> > > new default behavior where test failures will block merging, ensuring
> > 
> > that
> > 
> > > every commit keeps our code in a working state.
> > > 
> > > 
> > > ## Key Details
> > > 
> > > ### New Opt-Out Option
> > > 
> > > We've introduced a new `allow-failing-tests-on` option for the
> > > `.kde-ci.yml` file for projects that need temporary flexibility. If your
> > > project isn't ready for this change, you can opt out using this option.
> > > 
> > > Previously, the `require-passing-tests-on` option was used to make test
> > > failures fatal by adding it to the options in the `.kde-ci.yml` file
> > > like
> > > 
> > > this:
> > >     require-passing-tests-on: [ 'Linux', 'Windows' ]
> > > 
> > > If you need to opt out of the new default behavior, you can add the
> > > 
> > > `allow-failing-tests-on` option to your `.kde-ci.yml` file like this:
> > >     allow-failing-tests-on: [ 'Linux', 'Windows' ]
> > > 
> > > Full example of a `.kde-ci.yml` file with the new option:
> > > 
> > > ```yaml
> > > # SPDX-FileCopyrightText: None
> > > # SPDX-License-Identifier: CC0-1.0
> > > 
> > > Dependencies:
> > > - 'on': ['@all']
> > > 
> > >   'require':
> > >     'frameworks/kcoreaddons': '@stable-kf6'
> > > 
> > > Options:
> > >   allow-failing-tests-on: [ 'Windows' ]
> > > 
> > > ```
> > > 
> > > The new opt-out option is available now, and the default behavior has
> > 
> > **not
> > 
> > > yet changed**. We will have a transition period to allow projects to
> > 
> > adjust
> > 
> > > before implementing the new default.
> > > 
> > > 
> > > ### Transition Period
> > > 
> > > There will be a 6-week transition period to allow projects to adjust
> > 
> > before
> > 
> > > making test failures fatal by default. During this time we will:
> > >   - Monitor adoption and feedback.
> > >   - Assist projects that haven't been adjusted yet, especially those
> > 
> > under
> > 
> > > community maintenance.
> > > 
> > > 
> > > ### Final Steps
> > > 
> > > After the transition period, test failures will become fatal by default.
> > > 
> > > Additionally:
> > >   - The legacy `require-passing-tests-on` option will be removed.
> > >   - We will communicate the change to the community via the mailing list
> > > 
> > > and Discourse.
> > > 
> > > 
> > > ### What You Need to Do
> > > 
> > > We encourage all maintainers to review and update their CI
> > > configurations
> > > accordingly. Your proactive adjustments and feedback will help ensure a
> > > smooth transition and maintain the high quality of KDE projects.
> > > 
> > > If your project needs to allow a platform to continue failing tests
> > > temporarily, add the `allow-failing-tests-on` option to your
> > 
> > `.kde-ci.yml`
> > 
> > > file with the appropriate platforms listed.
> > > 
> > > If some tests are failing because the Windows CI can't handle certain
> > > GUI
> > > tests, prefer conditionally skipping only those tests on Windows CI runs
> > 
> > by
> > 
> > > adding the following to the problematic tests:
> > > 
> > > ```cpp
> > > if (qEnvironmentVariableIntValue("KDECI_CANNOT_CREATE_WINDOWS")) {
> > > 
> > >     QSKIP("KDE CI can't create a window on this platform, skipping this
> > > 
> > > test");
> > > }
> > > ```
> > > 
> > > If you have any questions or need support during this process, reply
> > > here
> > > or in the accompanying Discourse thread:
> > > 
> > > https://discuss.kde.org/t/test-failures-fatal-by-default-in-kde-ci/31670
> > > 
> > > 
> > > ## TLDR
> > > 
> > > Test failures will be fatal by default in KDE CI. Use
> > > `allow-failing-tests-on` to opt out temporarily. Review and update your
> > 
> > CI
> > 
> > > configurations to ensure a smooth transition. Reach out for support if
> > > needed.
> > 
> > Two questions:
> > 
> > A) Where was this discussed? I am relatively involved in getting people to
> > get
> > tests passing, etc. and had no idea this was being worked on, if i had
> > known
> > it would have saved me creating all the MRs i created recently for KDE
> > Gear to
> > enable forcing tests to pass.
> > 
> > B) Who is going to update all the repositories in KDE Gear (or elsewhere
> > but
> > KDE Gear is my particular worry) that are community maintained (i.e. don't
> > have a clear maintainer)?
> > 
> > Cheers,
> > 
> >   Albert
> >   
> > > ---
> > > 
> > > Thank you for your attention and commitment to maintaining robust,
> > > high-quality code.
> > > 
> > > Cheers,
> > > Kristen