Gitlab CI: failed unit tests vs. currently passing CI

[Ben Cooksley]

On Sun, Jan 23, 2022 at 12:38 PM Albert Astals Cid <aacid at kde.org> wrote:

> El diumenge, 23 de gener de 2022, a les 0:09:23 (CET), Ben Cooksley va
> escriure:
> > On Sun, Jan 23, 2022 at 11:29 AM Albert Astals Cid <aacid at kde.org>
> wrote:
> >
> > > El dissabte, 22 de gener de 2022, a les 6:11:29 (CET), Ben Cooksley va
> > > escriure:
> > > > EXCLUDE_DEPRECATED_BEFORE_AND_ATOn Sat, Jan 22, 2022 at 1:31 PM
> Friedrich
> > > > W. H. Kossebau <kossebau at kde.org> wrote:
> > > >
> > > > > Hi,
> > > >
> > > >
> > > > > seems that Gitlab CI is currently configured to show the green
> > > "Success"
> > > > > checkmark for pipeline runs even if unit tests are failing.
> > > > >
> > > >
> > > > That is correct, only compilation or other internal failures cause
> the
> > > > build to show a failure result.
> > > >
> > > >
> > > > > Reasons seems to be that there Gitlab only knows Yay or Nay,
> without
> > > the
> > > > > warning state level known from Jenkins.
> > > > >
> > > >
> > > > Also correct.
> > > >
> > > >
> > > > > And given that quite some projects (sadly) maintain a few long-time
> > > > > failing
> > > > > unit tests, having the pipeline fail on unit tests seems to have
> been
> > > seen
> > > > > as
> > > > > too aggressive
> > > >
> > > >
> > > > Correct again.
> > > >
> > > >
> > > > >
> > > > >
> > > > > This of course harms the purpose of the unit tests, when their
> failures
> > > > > are
> > > > > only noticed weeks later, not e.g. at MR discussion time.
> > > > >
> > > >
> > > > Gitlab does note changes in the test suite as can currently be seen
> on
> > > > https://invent.kde.org/frameworks/kio/-/merge_requests/708
> > > > Quoting the page:  "Test summary contained 33 failed and 16 fixed
> test
> > > > results out of 205 total tests"
> > > >
> > > > It does the same thing for Code Quality - "Code quality scanning
> detected
> > > > 51 changes in merged results"
> > >
> > > Don't want to derail the confirmation, but those results are terrible,
> > > they always say things changed in places not touched by the code of
> the MR,
> > > any idea why?
> > >
> >
> > Unfortunately not - my only guess would be that cppcheck reports results
> > slightly differently which Gitlab has issues interpreting.
>
> Can we just disable it?
>

Various things can be configured on a per-project basis. cppcheck is one of
them.
See
https://invent.kde.org/sysadmin/ci-utilities/-/blob/master/config-template.yml#L21


>
> Look at the results here
> https://invent.kde.org/graphics/okular/-/merge_requests/544
>
> Major - Either the condition 'printDialog' is redundant or there is
> possible null pointer dereference: printDialog. (CWE-476)
> in part/part.cpp:3341
>
> Fixed: Major - Either the condition 'printDialog' is redundant or there is
> possible null pointer dereference: printDialog. (CWE-476)
> in part/part.cpp:3340
>
> gitlab my friend, don't you think that maybe, just maybe this is the same
> code and you shouldn't complain to me about it since the only change to
> that file is 3000 lines away from it?
>

This is possibly cppcheck's fault, but yes not terribly good work there on
fuzzing for line changes.


>
> I find it confusing, it always makes me sad lowering my productivity.
>
> Cheers,
>   Albert
>
>
>
Cheers,
Ben
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.kde.org/pipermail/kde-frameworks-devel/attachments/20220123/47146f57/attachment-0001.htm>