Gitlab CI: failed unit tests vs. currently passing CI

[Ben Cooksley]

EXCLUDE_DEPRECATED_BEFORE_AND_ATOn Sat, Jan 22, 2022 at 1:31 PM Friedrich
W. H. Kossebau <kossebau at kde.org> wrote:

> Hi,


> seems that Gitlab CI is currently configured to show the green "Success"
> checkmark for pipeline runs even if unit tests are failing.
>

That is correct, only compilation or other internal failures cause the
build to show a failure result.


> Reasons seems to be that there Gitlab only knows Yay or Nay, without the
> warning state level known from Jenkins.
>

Also correct.


> And given that quite some projects (sadly) maintain a few long-time
> failing
> unit tests, having the pipeline fail on unit tests seems to have been seen
> as
> too aggressive


Correct again.


>
>
> This of course harms the purpose of the unit tests, when their failures
> are
> only noticed weeks later, not e.g. at MR discussion time.
>

Gitlab does note changes in the test suite as can currently be seen on
https://invent.kde.org/frameworks/kio/-/merge_requests/708
Quoting the page:  "Test summary contained 33 failed and 16 fixed test
results out of 205 total tests"

It does the same thing for Code Quality - "Code quality scanning detected
51 changes in merged results"


>
> Seeing how at least in KDE Frameworks first regressions sneaked in without
> someone noticing (nobody looks at logs when the surface shows a green
> checkmark, e.g. kcoreaddons, kwidgetsaddons, kio, purpose, krunner on
> openSUSE
> and possibly more have regressed in recent weeks, see build.kde.org) this
> should be something to deal with better, right?


> Bhushan gave two first ideas just now on #kde-sysadmin:
> > Well we can add a switch that repos can commit to saying test failure is
> build failure
> > Another alternative is we use bot to write a comment on MR
>
> IMHO, to give unit tests the purpose they have, we should by default to
> let
> test failures be build failures. And have projects opt out if they need to
> have some unit tests keep failing, instead of e.g. tagging them with
> expected
> failures or handling any special environment they run into on the CI.
>
> Your opinions?
>

The switch will need to be around the other way i'm afraid as there are
simply too many projects with broken tests right now.
The best place for that switch will be in .kde-ci.yml.

My only concern however would be abuse of this switch, much in the way that
certain projects abuse EXCLUDE_DEPRECATED_BEFORE_AND_AT.
The last thing we would want would be for people to flip this switch and
then leave their CI builds in a failing state - meaning that actual
compilation failures would be missed (and then lead to CI maintenance
issues)

Thoughts on that?


>
> Cheers
> Friedrich
>

Cheers,
Ben
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.kde.org/pipermail/kde-frameworks-devel/attachments/20220122/7fe26132/attachment-0001.htm>