<div dir="ltr"><div dir="ltr">On Mon, Mar 14, 2022 at 4:44 PM Eduardo de Souza Cruz <<a href="mailto:eduardo.cruz@kdemail.net">eduardo.cruz@kdemail.net</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Hi,</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Regarding the krunner timer-based test which I authored:</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
We could just delete the few lines that are enforcing those upper-limit timeouts and add some comments explaining that in real life those timeouts should have been met, I don't think we need to delete the entire test. I was explicitly asked to write a test
when I submitted the MR to enforce that this functionally wouldn't regress in the future.<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Also, I'm wondering... if we happen to have a compilation directive that's #defined in this CI environment only, or some framework function that returns weather we are in the CI environment or not, we could put some #ifndef (or if's) on those few upper-limit
timeout lines and avoid compiling/running just those lines in the CI environment. I'm not familiar with this environment so I don't know if there is such a thing, I'm just wondering...</div></div></blockquote><div><br></div><div>Not at this time i'm afraid. </div><div><br></div><div>There are some environment variables you can look to but those differ between our Jenkins and Gitlab setups.</div><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
I'm available to submit a quick MR if you'd like me to, just give me some directions for what you'd like me to do.<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
[],</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Eduardo<br></div></div></blockquote><div><br></div><div>Cheers,</div><div>Ben</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
</div>
<div id="gmail-m_-4471354558314000080appendonsend"></div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<hr style="display:inline-block;width:98%">
<div id="gmail-m_-4471354558314000080divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>From:</b> Ben Cooksley <<a href="mailto:bcooksley@kde.org" target="_blank">bcooksley@kde.org</a>><br>
<b>Sent:</b> Sunday, March 13, 2022 1:53 PM<br>
<b>To:</b> KDE Frameworks <<a href="mailto:kde-frameworks-devel@kde.org" target="_blank">kde-frameworks-devel@kde.org</a>><br>
<b>Cc:</b> <a href="mailto:eduardo.cruz@kdemail.net" target="_blank">eduardo.cruz@kdemail.net</a> <<a href="mailto:eduardo.cruz@kdemail.net" target="_blank">eduardo.cruz@kdemail.net</a>><br>
<b>Subject:</b> Re: Unit tests all pass in Jenkins on Linux</font>
<div> </div>
</div>
<div>
<div dir="ltr">
<div dir="ltr">On Mon, Mar 14, 2022 at 4:40 AM David Faure <<a href="mailto:faure@kde.org" target="_blank">faure@kde.org</a>> wrote:<br>
</div>
<div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
After the recent discussions on state of CI, I fixed the last unittest failures (kio, purpose... + apol fixed ECM) so that<br>
<a href="https://build.kde.org/job/Frameworks/view/Platform%20-%20SUSEQt5.15/" rel="noreferrer" target="_blank">https://build.kde.org/job/Frameworks/view/Platform%20-%20SUSEQt5.15/</a>
<br>
is all green^H^Hblue again.<br>
Please keep it that way!<br>
</blockquote>
<div><br>
</div>
<div>Thanks for looking into and fixing all of these David.</div>
<div> </div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
Note however that<br>
<br>
* kwayland has a flaky test:<br>
<br>
<a href="https://build.kde.org/job/Frameworks/view/Platform%20-%20SUSEQt5.15/job/kwayland/job/kf5-qt5%20SUSEQt5.15/171/testReport/junit/projectroot.autotests/client/kwayland_testDataDevice/" rel="noreferrer" target="_blank">https://build.kde.org/job/Frameworks/view/Platform%20-%20SUSEQt5.15/job/kwayland/job/kf5-qt5%20SUSEQt5.15/171/testReport/junit/projectroot.autotests/client/kwayland_testDataDevice/</a><br>
<br>
FAIL! : TestDataDevice::testReplaceSource() Compared values are not the same<br>
Actual (selectionOfferedSpy.count()): 1<br>
Expected (2) : 2<br>
Loc: [autotests/client/test_datadevice.cpp(557)]<br>
<br>
Who can look at this one? git log mostly shows Martin Flöser <<a href="mailto:mgraesslin@kde.org" target="_blank">mgraesslin@kde.org</a>><br>
who I think isn't active anymore?<br>
</blockquote>
<div><br>
</div>
<div>Not sure if it applies to KWayland as well, but I know that KWin has load sensitive tests (which is why the Gitlab .kde-ci.yml files support the flag <span style="color:rgb(0,0,0);font-family:monospace">tests-load-sensitive</span>)</div>
<div>If this test appears to be flaky, then it is quite possible that it is load sensitive as well.</div>
<div> </div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
* krunner has a flaky test [2] because it measures time spent and expects small values like 65ms<br>
(I changed that one to 100ms), 250ms, 300ms. With only 10% safety margins. On a busy CI system,<br>
this is bound to fail regularly, even with bigger safety margins. In my experience this kind of test<br>
is just not possible (we're not running on a real time OS), I vote for removing the test.<br>
CC'ing Eduardo.<br>
<br>
<a href="https://build.kde.org/job/Frameworks/view/Platform%20-%20SUSEQt5.15/job/krunner/job/kf5-qt5%20SUSEQt5.15/325/testReport/junit/projectroot/autotests/runnermanagertest/" rel="noreferrer" target="_blank">https://build.kde.org/job/Frameworks/view/Platform%20-%20SUSEQt5.15/job/krunner/job/kf5-qt5%20SUSEQt5.15/325/testReport/junit/projectroot/autotests/runnermanagertest/</a></blockquote>
<div><br>
</div>
<div>Yes, that will definitely fail more often than not - your only way to make sure tests like this pass on our CI system is to set tests-load-sensitive=True (in Gitlab CI)</div>
<div>Note however that option should be avoided where possible as it means your build will stop and wait for load to fall to low levels before proceeding with running tests - which blocks a CI worker slot from being used by another project.</div>
<div><br>
</div>
<div>I'd also be in favour of removing this test.</div>
<div><br>
</div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
<br>
-- <br>
David Faure, <a href="mailto:faure@kde.org" target="_blank">faure@kde.org</a>, <a href="http://www.davidfaure.fr" rel="noreferrer" target="_blank">
http://www.davidfaure.fr</a><br>
Working on KDE Frameworks 5<br>
<br>
<br>
<br>
</blockquote>
<div><br>
</div>
<div>Cheers,</div>
<div>Ben </div>
</div>
</div>
</div>
</div>
</blockquote></div></div>