Fixing tests failing on CI but not on local system

[Amber Schenck]

On 4/23/25 2:53 AM, Stefano Crocco wrote:
> On martedì 22 aprile 2025 20:57:53 Ora legale dell’Europa centrale Ben
> Cooksley wrote:
>> On Wed, Apr 23, 2025 at 5:24 AM Stefano Crocco <stefano.crocco at alice.it>
>>
>> wrote:
>>> Hello to everyone,
>>
>> Hi Stefano,
> 
> Hello Ben,
>>
>>> recently, I've being trying to fix failing Konqueror tests, and in doing
>>> so I'm
>>> facing some issue with the CI tests which I don't know how to solve.
>>>
>>> The first issue is that one test passes on my system but fails when run by
>>> the
>>> CI (it's the konqview test at [1]). I couldn't spot anything in the code
>>> which
>>> would be likely to be different on the CI than on my system. What do you
>>> think
>>> is the best way to try and fix this test? Should I try and add some
>>> debugging
>>> statement in the tests, then run it again on the CI to try to pinpoint the
>>> cause of the failure? I wouldn't want to waste CI resources, but I have no
>>> better idea.
>>
>> This test looks like it might be a KXMLGUI initialisation failure:
>>
>> QWARN  : KonqViewTest::textThenHtml() kf.xmlgui: cannot find .rc file
>> "konqueror.rc" for component "konqviewtest"
>> QWARN  : KonqViewTest::textThenHtml() kf.xmlgui: cannot find .rc file
>> "konqueror.rc" for component "konqviewtest"
>>
>> It is probably masked on your local system because there will be old files
>> / configuration / etc present.
>> Might be worth creating a new user account, doing a build of Konqueror
>> alone there (without logging in or running Konqueror normally there) and
>> seeing if the unit test fails there as that is basically what CI does.
> 
> I don't think this can be the difference because I get the same message on my
> system where the test succeeds (because the konqueror.rc file is installed in
> the konqueror directory, while the KXmlGui frameworks expects it in the
> konqviewtest directory since that's the name of the component) . Just to be
> sure, I tried following your suggestion but nothing changed and the test
> succeeded.
> 

We were writing tests for code that would write to /etc in production 
and of course didn't want that to happen in the test case, so we used 
the sandbox tool from gentoo[1] to ensure that it would only read/write 
from the test data dir instead of /etc. It also conveniently showed that 
it was reading from ~/.config/ which could have led to inconsistent 
results on different systems. That at least will show a stack trace 
where in the code the read is coming from, which is a hint to start 
tracking how to prevent it from happening in tests. For our tests the 
solution was to use QTEST_APPLESS_MAIN instead of QTEST_MAIN.

Happy Testing and Good Luck!
-Amber


1: https://wiki.gentoo.org/wiki/Project:Sandbox>>> A similar issue 
regards tests failing on FreeBSD but not on Linux: how
>>> should
>>> I proceed, since that I have no FreeBSD machine available and I'm not
>>> familiar
>>> enough with it to try setting up a virtual machine?
>>
>> In the not too distant future once we have VM based CI you'd be able to
>> download the VM image the CI system uses.
>> This is weeks, not months, away at this point.
> 
> This should be very useful. Thanks for the information.
> 
>>
>>> Lastly, there's a test (konqviewmgrtest at [2]) which is marked as failed
>>> by
>>> the CI but, looking at the output it reports, seems successful to me (I
>>> can't
>>> see any mention of failing tests or errors in the output). I attach a file
>>> with
>>> the output of the test. My only guess is that the word "failed" which
>>> appears
>>> in lines 3 and 28 but which is unrelated to the test itself may be
>>> confusing
>>> the CI. Do you think it is possible? Otherwise, do you have any idea about
>>> why
>>> the test could be marked as failed?
>>
>> Reading the full build log reveals why that happened:
>>
>> 4/12 Test #4: konqviewmgrtest ...................***Timeout 60.12 sec
>>
>> My guess would be that it is popping up a message box as something didn't
>> quite work out when it tries to activate the link:
>>
>> QDEBUG : ViewMgrTest::testLinkedViews() ACTIVATING LINK
> 
> Thanks for pointing this out. I didn't think of looking at the full output.
> I'll try to find out could be producing a message box, but it won't be easy.
> 
>>
>>> Thanks in advance
>>>
>>> Stefano
>>
>> Hope that gives you some ideas!
>>
>> Cheers,
>> Ben
> 
> Thanks
> 
> Stefano
> 
> 

-- 
Attached is my PGP public key.
Primary key fingerprint: 4407 5FB3 3665 0970 3B75 CD31 7DA1 7F4D AC46 7943

If you have a PGP key (and a minute to spare)
please send it in reply to this email.

If you have no idea what PGP is, feel free
to ignore all this gobbledegook.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_0x7DA17F4DAC467943.asc
Type: application/pgp-keys
Size: 4483 bytes
Desc: OpenPGP public key
URL: <http://mail.kde.org/pipermail/kde-devel/attachments/20250423/dcd24686/attachment.key>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: OpenPGP digital signature
URL: <http://mail.kde.org/pipermail/kde-devel/attachments/20250423/dcd24686/attachment.sig>