Unit tests all pass in Jenkins on Linux

[Ben Cooksley]

On Mon, Mar 14, 2022 at 4:40 AM David Faure <faure at kde.org> wrote:

> After the recent discussions on state of CI, I fixed the last unittest
> failures (kio, purpose... + apol fixed ECM) so that
> https://build.kde.org/job/Frameworks/view/Platform%20-%20SUSEQt5.15/
> is all green^H^Hblue again.
> Please keep it that way!
>

Thanks for looking into and fixing all of these David.


>
> Note however that
>
> * kwayland has a flaky test:
>
>
> https://build.kde.org/job/Frameworks/view/Platform%20-%20SUSEQt5.15/job/kwayland/job/kf5-qt5%20SUSEQt5.15/171/testReport/junit/projectroot.autotests/client/kwayland_testDataDevice/
>
> FAIL!  : TestDataDevice::testReplaceSource() Compared values are not the
> same
>    Actual   (selectionOfferedSpy.count()): 1
>    Expected (2)                          : 2
>    Loc: [autotests/client/test_datadevice.cpp(557)]
>
> Who can look at this one? git log mostly shows Martin Flöser <
> mgraesslin at kde.org>
> who I think isn't active anymore?
>

Not sure if it applies to KWayland as well, but I know that KWin has load
sensitive tests (which is why the Gitlab .kde-ci.yml files support the flag
tests-load-sensitive)
If this test appears to be flaky, then it is quite possible that it is load
sensitive as well.


>
> * krunner has a flaky test [2] because it measures time spent and expects
> small values like 65ms
> (I changed that one to 100ms), 250ms, 300ms. With only 10% safety margins.
> On a busy CI system,
> this is bound to fail regularly, even with bigger safety margins. In my
> experience this kind of test
> is just not possible (we're not running on a real time OS), I vote for
> removing the test.
> CC'ing Eduardo.
>
>
> https://build.kde.org/job/Frameworks/view/Platform%20-%20SUSEQt5.15/job/krunner/job/kf5-qt5%20SUSEQt5.15/325/testReport/junit/projectroot/autotests/runnermanagertest/


Yes, that will definitely fail more often than not - your only way to make
sure tests like this pass on our CI system is to
set tests-load-sensitive=True (in Gitlab CI)
Note however that option should be avoided where possible as it means your
build will stop and wait for load to fall to low levels before proceeding
with running tests - which blocks a CI worker slot from being used by
another project.

I'd also be in favour of removing this test.


>
> --
> David Faure, faure at kde.org, http://www.davidfaure.fr
> Working on KDE Frameworks 5
>
>
>
>
Cheers,
Ben
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.kde.org/pipermail/kde-frameworks-devel/attachments/20220314/98e82629/attachment-0001.htm>