Skyrocketing test run times

[Ben Cooksley]

On Wed, Feb 4, 2026 at 10:05 PM Vlad Zahorodnii <vlad.zahorodnii at kde.org>
wrote:

> On 2/4/26 10:26 AM, Ben Cooksley wrote:
>
> On Wed, Feb 4, 2026 at 9:07 PM Ben Cooksley <bcooksley at kde.org> wrote:
>
>> On Wed, Feb 4, 2026 at 8:25 PM Vlad Zahorodnii <vlad.zahorodnii at kde.org>
>> wrote:
>>
>>> Hello,
>>>
>>
>> HI Vlad,
>>
>>
>>>
>>> Over approximately the past weekend something happened in our CI and now
>>> it takes quite long time for tests to run. For example, in kwin, we have
>>> a test that used to run for about 20 seconds, and now it takes about 5
>>> or so minutes to finish running. Speaking for kwin, there were no
>>> changes that could increase test run times so dramatically.
>>>
>>> January 26th:
>>>
>>>          Start  61: kwin-testOutputChanges
>>>   61/158 Test  #61: kwin-testOutputChanges
>>> .............................   Passed   19.36 sec
>>>
>>> January 29th:
>>>
>>>          Start  61: kwin-testOutputChanges
>>>   61/158 Test  #61: kwin-testOutputChanges
>>> .............................   Passed   43.93 sec
>>>
>>> January 30th:
>>>
>>>          Start  61: kwin-testOutputChanges
>>>   61/158 Test  #61: kwin-testOutputChanges
>>> .............................   Passed   45.91 sec
>>>
>>> Februrary 3rd:
>>>
>>>          Start  61: kwin-testOutputChanges
>>>   61/158 Test  #61: kwin-testOutputChanges
>>> .............................   Passed  254.19 sec
>>>
>>> FreeBSD appears to be fine.
>>>
>>> We suspect that test run times blew up due to enabling LSAN in various
>>> libraries (kwin itself has no LSAN enabled yet). The issue doesn't
>>> appear to be specific to only kwin, people reported that they've seen
>>> similar issues in other projects too. Maybe something else happened to
>>> CI that sysadmins will be able to clarify.
>>>
>>
>> Nothing else happened to CI recently aside from the enablement of LSAN.
>>
>> The underlying SUSE images were for Qt 6.10 at least last rebuilt on
>> January 25th, which is well before your "last good" date.
>>
>> The only change to CI between January 30th and February 3rd
>> was fast_unwind_on_malloc=0 being added by default, even though it is
>> primarily for the benefit of LSAN.
>> I've now made changes to only set fast_unwind_on_malloc=0 if LSAN is
>> explicitly enabled for a repository - hard to tell if that will fix the
>> issue though as KWin takes a while to build.
>>
>
> For the record, as per https://invent.kde.org/plasma/kwin/-/jobs/3958895
> which completed moments ago:
>
> That branch contained some other things that could interfere with test
> results. I started https://invent.kde.org/plasma/kwin/-/jobs/3959078 and
> yeah it looks like test run times are back to the Jan 29-30th level.
>
> So, it seems like fast_unwind_on_malloc=0 is the culprit then?
>

Yes, quoting from https://source.android.com/docs/security/test/asan

[quote]
ASan uses a fast, frame-pointer-based unwinder to record a stack trace for
every memory allocation and deallocation event in the program. Most of
Android is built without frame pointers. As a result, you often get only
one or two meaningful frames. To fix this, either rebuild the library with
ASan (recommended!), or with:

LOCAL_CFLAGS:=-fno-omit-frame-pointer
LOCAL_ARM_MODE:=arm

Or set ASAN_OPTIONS=fast_unwind_on_malloc=0 in the process environment. The
latter can be very CPU-intensive, depending on the load.
[/quote]

That would align with what we're seeing here

Regards,
> Vlad
>

Cheers,
Ben

>
>         Start  61: kwin-testOutputChanges
>  61/158 Test  #61: kwin-testOutputChanges .............................
> Passed   19.58 sec
>
>
>>
>>
>>>
>>> Either way, the current state of CI is not great. Hypothetically, test
>>> timeouts can be increased but QSignalSpy's have hardcoded timeouts that
>>> can be too low for the current CI. And in case of kwin, 5 minutes for a
>>> test is simply not a workable thing.
>>>
>>> Regards,
>>> Vlad
>>>
>>
>> Thanks,
>> Ben
>>
>
> Cheers,
> Ben
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.kde.org/pipermail/plasma-devel/attachments/20260204/91507f76/attachment.htm>