KF 5.91: 24 modules with failing unit tests (Re: Please fix failing unit tests with Windows platform)

[Ben Cooksley]

On Mon, Feb 7, 2022 at 10:56 PM Christoph Cullmann (cullmann.io) <
christoph at cullmann.io> wrote:

> On 2022-02-07 10:35, Ben Cooksley wrote:
> > On Sun, Feb 6, 2022 at 10:40 PM Friedrich W. H. Kossebau
> > <kossebau at kde.org> wrote:
> >
> >> Am Montag, 24. Januar 2022, 01:06:40 CET schrieb Friedrich W. H.
> >> Kossebau:
> >>> Hi,
> >>>
> >>> since a long time there are lots of failing unit tests across
> >> multiple
> >>> repositories. Could the Windows platform maintainers/stakeholders
> >> please
> >>> look soonish into either fixing those tests or properly marking
> >> them as
> >>> expected to fail, so the resources the KDE CI spends on running
> >> the tests
> >>> every hour, day and week make some sense again, as well as having
> >> something
> >>> usable to diff results again, to notice any new regressions?
> >>>
> >>> Please see
> >>>
> >>>
> >>
> >
> https://build.kde.org/job/Frameworks/view/Platform%20-%20WindowsMSVCQt5.15/
> >>> (best sort by "S" build status to get a list what need
> >>
> >> And those who believe in the broken windows theory also would claim
> >> this
> >> slacking now resulted in the regressions in the openSUSE builds,
> >> where 5
> >> modules now have failing unit tests at time of release tagging, when
> >> it once
> >> was 0 thanks to hard work of David F. and others. :(
> >>
> >> Is it time to remove
> >> https://community.kde.org/Frameworks/
> >> Policies#Frameworks_CI_failures_are_treated_as_stop_the_line_events
> >> because seemingly this is just old dead pixels on a web page and not
> >> the
> >> spirit these days?
> >
> > Not sure that is the ideal outcome here - preferrably our tests would
> > continue to all pass.
> >
> > I know some tests on certain platforms have been flaky and switch
> > between failing/passing - are we sure that isn't the driver of people
> > ignoring the results?
>
> One thing that sometimes lead me to ignore stuff with KTextEditor is
> that the UI
> tests are often very unstable.
>
> e.g. they just fail for me locally but then work perfectly in the CI or
> the other
> way around.
>
> Not sure how to improve that.
>

Interesting to note that it is GUI/UI tests that are causing issues. I
would have thought that the setup on the CI for those would be a carbon
copy almost every time which makes these failures interesting.
Out of curiosity, what are the tests trying to accomplish and where is it
failing?


>
> For all non-UI tests naturally no such problems exist for KTextEditor
> and they are easy to keep
> working.
>
> KSyntaxHighlighting only has non-UI test and that is very easy to keep
> in a consistent shape.
>
> Greetings
> Christoph
>

Thanks,
Ben


>
> >
> > Cheers,
> > Ben
> >
> >> Friedrich
>
> --
> Ignorance is bliss...
> https://cullmann.io | https://kate-editor.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.kde.org/pipermail/kde-frameworks-devel/attachments/20220207/254f39b2/attachment.htm>