Test Failures Fatal by Default in KDE CI

[Kristen McWilliam]

On Wed, Mar 19, 2025 at 11:30 AM Albert Astals Cid <aacid at kde.org> wrote:

> El dimecres, 19 de març del 2025, a les 16:22:40 (Hora estàndard d’Europa
> central), Kristen McWilliam va escriure:
> > A) Discussion was initially in
> > https://invent.kde.org/sysadmin/ci-utilities/-/merge_requests/296 and
> then
> > moved to https://invent.kde.org/sysadmin/ci-utilities/-/issues/37 for
> > overall tracking. Sorry to hear you did that extra work; we're hoping to
> > consolidate the effort with this. :)
> >
> > B) No one has made any specific commitments to helping with this, but I
> > know I for one will be trying my best to help where needed, such as the
> > community maintained repos mentioned. Any assistance with that is
> obviously
> > very welcome.
>
> Thanks :)
>
> Cheers,
>   Albert
>
> P.S: Please make sure you also answer to the list :)
>

My bad, didn't realize I was hitting the wrong reply button. Thanks for
pointing that out! :)


>
> >
> >
> > Cheers,
> > Kristen
> >
> > On Wed, Mar 19, 2025 at 10:48 AM Albert Astals Cid <aacid at kde.org>
> wrote:
> > > El dilluns, 17 de març del 2025, a les 14:54:24 (Hora estàndard
> d’Europa
> > >
> > > central), Kristen McWilliam va escriure:
> > > > Hi all,
> > > >
> > > > I'm excited to share an important update to our Continuous
> Integration
> > > > system that will improve the stability and reliability of our
> software.
> > > > Starting soon, test failures will be fatal by default in KDE CI.
> > > >
> > > > Historically, making test failures fatal has been an opt-in feature.
> > > > This
> > > > allowed some projects to inadvertently overlook failing tests, which
> > >
> > > could
> > >
> > > > lead to undetected breakages. To address this, we are transitioning
> to a
> > > > new default behavior where test failures will block merging, ensuring
> > >
> > > that
> > >
> > > > every commit keeps our code in a working state.
> > > >
> > > >
> > > > ## Key Details
> > > >
> > > > ### New Opt-Out Option
> > > >
> > > > We've introduced a new `allow-failing-tests-on` option for the
> > > > `.kde-ci.yml` file for projects that need temporary flexibility. If
> your
> > > > project isn't ready for this change, you can opt out using this
> option.
> > > >
> > > > Previously, the `require-passing-tests-on` option was used to make
> test
> > > > failures fatal by adding it to the options in the `.kde-ci.yml` file
> > > > like
> > > >
> > > > this:
> > > >     require-passing-tests-on: [ 'Linux', 'Windows' ]
> > > >
> > > > If you need to opt out of the new default behavior, you can add the
> > > >
> > > > `allow-failing-tests-on` option to your `.kde-ci.yml` file like this:
> > > >     allow-failing-tests-on: [ 'Linux', 'Windows' ]
> > > >
> > > > Full example of a `.kde-ci.yml` file with the new option:
> > > >
> > > > ```yaml
> > > > # SPDX-FileCopyrightText: None
> > > > # SPDX-License-Identifier: CC0-1.0
> > > >
> > > > Dependencies:
> > > > - 'on': ['@all']
> > > >
> > > >   'require':
> > > >     'frameworks/kcoreaddons': '@stable-kf6'
> > > >
> > > > Options:
> > > >   allow-failing-tests-on: [ 'Windows' ]
> > > >
> > > > ```
> > > >
> > > > The new opt-out option is available now, and the default behavior has
> > >
> > > **not
> > >
> > > > yet changed**. We will have a transition period to allow projects to
> > >
> > > adjust
> > >
> > > > before implementing the new default.
> > > >
> > > >
> > > > ### Transition Period
> > > >
> > > > There will be a 6-week transition period to allow projects to adjust
> > >
> > > before
> > >
> > > > making test failures fatal by default. During this time we will:
> > > >   - Monitor adoption and feedback.
> > > >   - Assist projects that haven't been adjusted yet, especially those
> > >
> > > under
> > >
> > > > community maintenance.
> > > >
> > > >
> > > > ### Final Steps
> > > >
> > > > After the transition period, test failures will become fatal by
> default.
> > > >
> > > > Additionally:
> > > >   - The legacy `require-passing-tests-on` option will be removed.
> > > >   - We will communicate the change to the community via the mailing
> list
> > > >
> > > > and Discourse.
> > > >
> > > >
> > > > ### What You Need to Do
> > > >
> > > > We encourage all maintainers to review and update their CI
> > > > configurations
> > > > accordingly. Your proactive adjustments and feedback will help
> ensure a
> > > > smooth transition and maintain the high quality of KDE projects.
> > > >
> > > > If your project needs to allow a platform to continue failing tests
> > > > temporarily, add the `allow-failing-tests-on` option to your
> > >
> > > `.kde-ci.yml`
> > >
> > > > file with the appropriate platforms listed.
> > > >
> > > > If some tests are failing because the Windows CI can't handle certain
> > > > GUI
> > > > tests, prefer conditionally skipping only those tests on Windows CI
> runs
> > >
> > > by
> > >
> > > > adding the following to the problematic tests:
> > > >
> > > > ```cpp
> > > > if (qEnvironmentVariableIntValue("KDECI_CANNOT_CREATE_WINDOWS")) {
> > > >
> > > >     QSKIP("KDE CI can't create a window on this platform, skipping
> this
> > > >
> > > > test");
> > > > }
> > > > ```
> > > >
> > > > If you have any questions or need support during this process, reply
> > > > here
> > > > or in the accompanying Discourse thread:
> > > >
> > > >
> https://discuss.kde.org/t/test-failures-fatal-by-default-in-kde-ci/31670
> > > >
> > > >
> > > > ## TLDR
> > > >
> > > > Test failures will be fatal by default in KDE CI. Use
> > > > `allow-failing-tests-on` to opt out temporarily. Review and update
> your
> > >
> > > CI
> > >
> > > > configurations to ensure a smooth transition. Reach out for support
> if
> > > > needed.
> > >
> > > Two questions:
> > >
> > > A) Where was this discussed? I am relatively involved in getting
> people to
> > > get
> > > tests passing, etc. and had no idea this was being worked on, if i had
> > > known
> > > it would have saved me creating all the MRs i created recently for KDE
> > > Gear to
> > > enable forcing tests to pass.
> > >
> > > B) Who is going to update all the repositories in KDE Gear (or
> elsewhere
> > > but
> > > KDE Gear is my particular worry) that are community maintained (i.e.
> don't
> > > have a clear maintainer)?
> > >
> > > Cheers,
> > >
> > >   Albert
> > >
> > > > ---
> > > >
> > > > Thank you for your attention and commitment to maintaining robust,
> > > > high-quality code.
> > > >
> > > > Cheers,
> > > > Kristen
>
>
>
>
>

-- 
Cheers,
Kristen
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.kde.org/pipermail/kde-devel/attachments/20250319/b41cb554/attachment-0001.htm>