Unit tests all pass in Jenkins on Linux

[Ben Cooksley]

On Mon, Mar 14, 2022 at 4:44 PM Eduardo de Souza Cruz <
eduardo.cruz at kdemail.net> wrote:

> Hi,
>
> Regarding the krunner timer-based test which I authored:
>
> We could just delete the few lines that are enforcing those upper-limit
> timeouts and add some comments explaining that in real life those timeouts
> should have been met, I don't think we need to delete the entire test. I
> was explicitly asked to write a test when I submitted the MR to enforce
> that this functionally wouldn't regress in the future.
>
> Also, I'm wondering... if we happen to have a compilation directive that's
> #defined in this CI environment only, or some framework function that
> returns weather we are in the CI environment or not, we could put some
> #ifndef (or if's) on those few upper-limit timeout lines and avoid
> compiling/running just those lines in the CI environment. I'm not familiar
> with this environment so I don't know if there is such a thing, I'm just
> wondering...
>

Not at this time i'm afraid.

There are some environment variables you can look to but those differ
between our Jenkins and Gitlab setups.


>
> I'm available to submit a quick MR if you'd like me to, just give me some
> directions for what you'd like me to do.
>
> [],
> Eduardo
>

Cheers,
Ben


>
> ------------------------------
> *From:* Ben Cooksley <bcooksley at kde.org>
> *Sent:* Sunday, March 13, 2022 1:53 PM
> *To:* KDE Frameworks <kde-frameworks-devel at kde.org>
> *Cc:* eduardo.cruz at kdemail.net <eduardo.cruz at kdemail.net>
> *Subject:* Re: Unit tests all pass in Jenkins on Linux
>
> On Mon, Mar 14, 2022 at 4:40 AM David Faure <faure at kde.org> wrote:
>
> After the recent discussions on state of CI, I fixed the last unittest
> failures (kio, purpose... + apol fixed ECM) so that
> https://build.kde.org/job/Frameworks/view/Platform%20-%20SUSEQt5.15/
> is all green^H^Hblue again.
> Please keep it that way!
>
>
> Thanks for looking into and fixing all of these David.
>
>
>
> Note however that
>
> * kwayland has a flaky test:
>
>
> https://build.kde.org/job/Frameworks/view/Platform%20-%20SUSEQt5.15/job/kwayland/job/kf5-qt5%20SUSEQt5.15/171/testReport/junit/projectroot.autotests/client/kwayland_testDataDevice/
>
> FAIL!  : TestDataDevice::testReplaceSource() Compared values are not the
> same
>    Actual   (selectionOfferedSpy.count()): 1
>    Expected (2)                          : 2
>    Loc: [autotests/client/test_datadevice.cpp(557)]
>
> Who can look at this one? git log mostly shows Martin Flöser <
> mgraesslin at kde.org>
> who I think isn't active anymore?
>
>
> Not sure if it applies to KWayland as well, but I know that KWin has load
> sensitive tests (which is why the Gitlab .kde-ci.yml files support the flag
> tests-load-sensitive)
> If this test appears to be flaky, then it is quite possible that it is
> load sensitive as well.
>
>
>
> * krunner has a flaky test [2] because it measures time spent and expects
> small values like 65ms
> (I changed that one to 100ms), 250ms, 300ms. With only 10% safety margins.
> On a busy CI system,
> this is bound to fail regularly, even with bigger safety margins. In my
> experience this kind of test
> is just not possible (we're not running on a real time OS), I vote for
> removing the test.
> CC'ing Eduardo.
>
>
> https://build.kde.org/job/Frameworks/view/Platform%20-%20SUSEQt5.15/job/krunner/job/kf5-qt5%20SUSEQt5.15/325/testReport/junit/projectroot/autotests/runnermanagertest/
>
>
> Yes, that will definitely fail more often than not - your only way to make
> sure tests like this pass on our CI system is to
> set tests-load-sensitive=True (in Gitlab CI)
> Note however that option should be avoided where possible as it means your
> build will stop and wait for load to fall to low levels before proceeding
> with running tests - which blocks a CI worker slot from being used by
> another project.
>
> I'd also be in favour of removing this test.
>
>
>
> --
> David Faure, faure at kde.org, http://www.davidfaure.fr
> Working on KDE Frameworks 5
>
>
>
>
> Cheers,
> Ben
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.kde.org/pipermail/kde-frameworks-devel/attachments/20220314/a8182de0/attachment.htm>