KF 5.91: 24 modules with failing unit tests (Re: Please fix failing unit tests with Windows platform)

[Christoph Cullmann (cullmann.io)]

On 2022-02-07 10:35, Ben Cooksley wrote:
> On Sun, Feb 6, 2022 at 10:40 PM Friedrich W. H. Kossebau
> <kossebau at kde.org> wrote:
> 
>> Am Montag, 24. Januar 2022, 01:06:40 CET schrieb Friedrich W. H.
>> Kossebau:
>>> Hi,
>>> 
>>> since a long time there are lots of failing unit tests across
>> multiple
>>> repositories. Could the Windows platform maintainers/stakeholders
>> please
>>> look soonish into either fixing those tests or properly marking
>> them as
>>> expected to fail, so the resources the KDE CI spends on running
>> the tests
>>> every hour, day and week make some sense again, as well as having
>> something
>>> usable to diff results again, to notice any new regressions?
>>> 
>>> Please see
>>> 
>>> 
>> 
> https://build.kde.org/job/Frameworks/view/Platform%20-%20WindowsMSVCQt5.15/
>>> (best sort by "S" build status to get a list what need
>> 
>> And those who believe in the broken windows theory also would claim
>> this
>> slacking now resulted in the regressions in the openSUSE builds,
>> where 5
>> modules now have failing unit tests at time of release tagging, when
>> it once
>> was 0 thanks to hard work of David F. and others. :(
>> 
>> Is it time to remove
>> https://community.kde.org/Frameworks/
>> Policies#Frameworks_CI_failures_are_treated_as_stop_the_line_events
>> because seemingly this is just old dead pixels on a web page and not
>> the
>> spirit these days?
> 
> Not sure that is the ideal outcome here - preferrably our tests would
> continue to all pass.
> 
> I know some tests on certain platforms have been flaky and switch
> between failing/passing - are we sure that isn't the driver of people
> ignoring the results?

One thing that sometimes lead me to ignore stuff with KTextEditor is 
that the UI
tests are often very unstable.

e.g. they just fail for me locally but then work perfectly in the CI or 
the other
way around.

Not sure how to improve that.

For all non-UI tests naturally no such problems exist for KTextEditor 
and they are easy to keep
working.

KSyntaxHighlighting only has non-UI test and that is very easy to keep 
in a consistent shape.

Greetings
Christoph

> 
> Cheers,
> Ben
> 
>> Friedrich

-- 
Ignorance is bliss...
https://cullmann.io | https://kate-editor.org