Gitlab CI: failed unit tests vs. currently passing CI

[Albert Astals Cid]

El diumenge, 23 de gener de 2022, a les 0:09:23 (CET), Ben Cooksley va escriure:
> On Sun, Jan 23, 2022 at 11:29 AM Albert Astals Cid <aacid at kde.org> wrote:
> 
> > El dissabte, 22 de gener de 2022, a les 6:11:29 (CET), Ben Cooksley va
> > escriure:
> > > EXCLUDE_DEPRECATED_BEFORE_AND_ATOn Sat, Jan 22, 2022 at 1:31 PM Friedrich
> > > W. H. Kossebau <kossebau at kde.org> wrote:
> > >
> > > > Hi,
> > >
> > >
> > > > seems that Gitlab CI is currently configured to show the green
> > "Success"
> > > > checkmark for pipeline runs even if unit tests are failing.
> > > >
> > >
> > > That is correct, only compilation or other internal failures cause the
> > > build to show a failure result.
> > >
> > >
> > > > Reasons seems to be that there Gitlab only knows Yay or Nay, without
> > the
> > > > warning state level known from Jenkins.
> > > >
> > >
> > > Also correct.
> > >
> > >
> > > > And given that quite some projects (sadly) maintain a few long-time
> > > > failing
> > > > unit tests, having the pipeline fail on unit tests seems to have been
> > seen
> > > > as
> > > > too aggressive
> > >
> > >
> > > Correct again.
> > >
> > >
> > > >
> > > >
> > > > This of course harms the purpose of the unit tests, when their failures
> > > > are
> > > > only noticed weeks later, not e.g. at MR discussion time.
> > > >
> > >
> > > Gitlab does note changes in the test suite as can currently be seen on
> > > https://invent.kde.org/frameworks/kio/-/merge_requests/708
> > > Quoting the page:  "Test summary contained 33 failed and 16 fixed test
> > > results out of 205 total tests"
> > >
> > > It does the same thing for Code Quality - "Code quality scanning detected
> > > 51 changes in merged results"
> >
> > Don't want to derail the confirmation, but those results are terrible,
> > they always say things changed in places not touched by the code of the MR,
> > any idea why?
> >
> 
> Unfortunately not - my only guess would be that cppcheck reports results
> slightly differently which Gitlab has issues interpreting.

Can we just disable it?

Look at the results here https://invent.kde.org/graphics/okular/-/merge_requests/544

Major - Either the condition 'printDialog' is redundant or there is possible null pointer dereference: printDialog. (CWE-476)
in part/part.cpp:3341

Fixed: Major - Either the condition 'printDialog' is redundant or there is possible null pointer dereference: printDialog. (CWE-476)
in part/part.cpp:3340

gitlab my friend, don't you think that maybe, just maybe this is the same code and you shouldn't complain to me about it since the only change to that file is 3000 lines away from it?

I find it confusing, it always makes me sad lowering my productivity.

Cheers,
  Albert