Skyrocketing test run times

[Ben Cooksley]

On Wed, Feb 4, 2026 at 9:07 PM Ben Cooksley <bcooksley at kde.org> wrote:

> On Wed, Feb 4, 2026 at 8:25 PM Vlad Zahorodnii <vlad.zahorodnii at kde.org>
> wrote:
>
>> Hello,
>>
>
> HI Vlad,
>
>
>>
>> Over approximately the past weekend something happened in our CI and now
>> it takes quite long time for tests to run. For example, in kwin, we have
>> a test that used to run for about 20 seconds, and now it takes about 5
>> or so minutes to finish running. Speaking for kwin, there were no
>> changes that could increase test run times so dramatically.
>>
>> January 26th:
>>
>>          Start  61: kwin-testOutputChanges
>>   61/158 Test  #61: kwin-testOutputChanges
>> .............................   Passed   19.36 sec
>>
>> January 29th:
>>
>>          Start  61: kwin-testOutputChanges
>>   61/158 Test  #61: kwin-testOutputChanges
>> .............................   Passed   43.93 sec
>>
>> January 30th:
>>
>>          Start  61: kwin-testOutputChanges
>>   61/158 Test  #61: kwin-testOutputChanges
>> .............................   Passed   45.91 sec
>>
>> Februrary 3rd:
>>
>>          Start  61: kwin-testOutputChanges
>>   61/158 Test  #61: kwin-testOutputChanges
>> .............................   Passed  254.19 sec
>>
>> FreeBSD appears to be fine.
>>
>> We suspect that test run times blew up due to enabling LSAN in various
>> libraries (kwin itself has no LSAN enabled yet). The issue doesn't
>> appear to be specific to only kwin, people reported that they've seen
>> similar issues in other projects too. Maybe something else happened to
>> CI that sysadmins will be able to clarify.
>>
>
> Nothing else happened to CI recently aside from the enablement of LSAN.
>
> The underlying SUSE images were for Qt 6.10 at least last rebuilt on
> January 25th, which is well before your "last good" date.
>
> The only change to CI between January 30th and February 3rd
> was fast_unwind_on_malloc=0 being added by default, even though it is
> primarily for the benefit of LSAN.
> I've now made changes to only set fast_unwind_on_malloc=0 if LSAN is
> explicitly enabled for a repository - hard to tell if that will fix the
> issue though as KWin takes a while to build.
>

For the record, as per https://invent.kde.org/plasma/kwin/-/jobs/3958895
which completed moments ago:

        Start  61: kwin-testOutputChanges
 61/158 Test  #61: kwin-testOutputChanges .............................
Passed   19.58 sec


>
>
>>
>> Either way, the current state of CI is not great. Hypothetically, test
>> timeouts can be increased but QSignalSpy's have hardcoded timeouts that
>> can be too low for the current CI. And in case of kwin, 5 minutes for a
>> test is simply not a workable thing.
>>
>> Regards,
>> Vlad
>>
>
> Thanks,
> Ben
>

Cheers,
Ben
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.kde.org/pipermail/plasma-devel/attachments/20260204/b4247099/attachment-0001.htm>