Gitlab CI: failed unit tests vs. currently passing CI

[Friedrich W. H. Kossebau]

Hi,

seems that Gitlab CI is currently configured to show the green "Success" 
checkmark for pipeline runs even if unit tests are failing.
Reasons seems to be that there Gitlab only knows Yay or Nay, without the 
warning state level known from Jenkins.
And given that quite some projects (sadly) maintain a few long-time failing 
unit tests, having the pipeline fail on unit tests seems to have been seen as 
too aggressive 

This of course harms the purpose of the unit tests, when their failures are 
only noticed weeks later, not e.g. at MR discussion time.

Seeing how at least in KDE Frameworks first regressions sneaked in without 
someone noticing (nobody looks at logs when the surface shows a green 
checkmark, e.g. kcoreaddons, kwidgetsaddons, kio, purpose, krunner on openSUSE 
and possibly more have regressed in recent weeks, see build.kde.org) this 
should be something to deal with better, right?

Bhushan gave two first ideas just now on #kde-sysadmin:
> Well we can add a switch that repos can commit to saying test failure is 
build failure
> Another alternative is we use bot to write a comment on MR

IMHO, to give unit tests the purpose they have, we should by default to let 
test failures be build failures. And have projects opt out if they need to 
have some unit tests keep failing, instead of e.g. tagging them with expected 
failures or handling any special environment they run into on the CI.

Your opinions?

Cheers
Friedrich