Fixing tests failing on CI but not on local system

[Stefano Crocco]

Hello to everyone,
recently, I've being trying to fix failing Konqueror tests, and in doing so I'm 
facing some issue with the CI tests which I don't know how to solve.

The first issue is that one test passes on my system but fails when run by the 
CI (it's the konqview test at [1]). I couldn't spot anything in the code which 
would be likely to be different on the CI than on my system. What do you think 
is the best way to try and fix this test? Should I try and add some debugging 
statement in the tests, then run it again on the CI to try to pinpoint the 
cause of the failure? I wouldn't want to waste CI resources, but I have no 
better idea.

A similar issue regards tests failing on FreeBSD but not on Linux: how should 
I proceed, since that I have no FreeBSD machine available and I'm not familiar 
enough with it to try setting up a virtual machine?

Lastly, there's a test (konqviewmgrtest at [2]) which is marked as failed by 
the CI but, looking at the output it reports, seems successful to me (I can't 
see any mention of failing tests or errors in the output). I attach a file with 
the output of the test. My only guess is that the word "failed" which appears 
in lines 3 and 28 but which is unrelated to the test itself may be confusing 
the CI. Do you think it is possible? Otherwise, do you have any idea about why 
the test could be marked as failed?

Thanks in advance

Stefano

[1] https://invent.kde.org/network/konqueror/-/pipelines/929647/test_report?
job_name=suse_tumbleweed_qt68
[2] https://invent.kde.org/network/konqueror/-/pipelines/936026/test_report?
job_name=suse_tumbleweed_qt69
-------------- next part --------------
********* Start testing of ViewMgrTest *********
Config: Using QtTest library 6.9.0, Qt 6.9.0 (x86_64-little_endian-lp64 shared (dynamic) release build; by GCC 14.2.1 20250220 [revision 9ffecde121af883b60bbe60d00425036bc873048]), opensuse-tumbleweed 20250417
[8363:8410:0422/163745.988433:ERROR:bus.cc(407)] Failed to connect to the bus: Failed to connect to socket /run/dbus/system_bus_socket: No such file or directory
kbuildsycoca6 running...
"applications.menu"  not found in  QList()
PASS   : ViewMgrTest::initTestCase()
QCRITICAL: ViewMgrTest::testCloseOtherTabs() "applications.menu"  not found in  QList()
PASS   : ViewMgrTest::testCloseOtherTabs()
PASS   : ViewMgrTest::testCloseTabsFast()
QDEBUG : ViewMgrTest::testCreateFirstView() QWidget(0x7bd4b4fd0a90, name="webenginepart") QRect(0,0 700x379)
QDEBUG : ViewMgrTest::testCreateFirstView() KonqFrame(0x7c64b4be4070) QRect(0,0 700x407)
QDEBUG : ViewMgrTest::testCreateFirstView() QStackedWidget(0x7bd4b4fd5e90, name="qt_tabwidget_stackedwidget") QRect(0,0 700x407)
QDEBUG : ViewMgrTest::testCreateFirstView() KonqFrameTabs(0x7c64b4be3fa0, name="kde_konq_tabwidget") QRect(0,73 700x407)
QDEBUG : ViewMgrTest::testCreateFirstView() KonqMainWindow(0x7b94b3e61260, name="MainWindow#1") QRect(0,0 700x480)
PASS   : ViewMgrTest::testCreateFirstView()
PASS   : ViewMgrTest::testEmptyWindow()
PASS   : ViewMgrTest::testRemoveFirstView()
PASS   : ViewMgrTest::testSplitView()
PASS   : ViewMgrTest::testSplitMainContainer()
QDEBUG : ViewMgrTest::testLinkedViews() SPLITTING
QDEBUG : ViewMgrTest::testLinkedViews() ACTIVATING LINK


System Out:

********* Start testing of ViewMgrTest *********
Config: Using QtTest library 6.9.0, Qt 6.9.0 (x86_64-little_endian-lp64 shared (dynamic) release build; by GCC 14.2.1 20250220 [revision 9ffecde121af883b60bbe60d00425036bc873048]), opensuse-tumbleweed 20250417
[8363:8410:0422/163745.988433:ERROR:bus.cc(407)] failed to connect to the bus: Failed to connect to socket /run/dbus/system_bus_socket: No such file or directory
kbuildsycoca6 running...
"applications.menu"  not found in  QList()
PASS   : ViewMgrTest::initTestCase()
QCRITICAL: ViewMgrTest::testCloseOtherTabs() "applications.menu"  not found in  QList()
PASS   : ViewMgrTest::testCloseOtherTabs()
PASS   : ViewMgrTest::testCloseTabsFast()
QDEBUG : ViewMgrTest::testCreateFirstView() QWidget(0x7bd4b4fd0a90, name="webenginepart") QRect(0,0 700x379)
QDEBUG : ViewMgrTest::testCreateFirstView() KonqFrame(0x7c64b4be4070) QRect(0,0 700x407)
QDEBUG : ViewMgrTest::testCreateFirstView() QStackedWidget(0x7bd4b4fd5e90, name="qt_tabwidget_stackedwidget") QRect(0,0 700x407)
QDEBUG : ViewMgrTest::testCreateFirstView() KonqFrameTabs(0x7c64b4be3fa0, name="kde_konq_tabwidget") QRect(0,73 700x407)
QDEBUG : ViewMgrTest::testCreateFirstView() KonqMainWindow(0x7b94b3e61260, name="MainWindow#1") QRect(0,0 700x480)
PASS   : ViewMgrTest::testCreateFirstView()
PASS   : ViewMgrTest::testEmptyWindow()
PASS   : ViewMgrTest::testRemoveFirstView()
PASS   : ViewMgrTest::testSplitView()
PASS   : ViewMgrTest::testSplitMainContainer()
QDEBUG : ViewMgrTest::testLinkedViews() SPLITTING
QDEBUG : ViewMgrTest::testLinkedViews() ACTIVATING LINK