Skyrocketing test run times

[Ben Cooksley]

On Thu, Feb 5, 2026 at 12:14 AM Nicolas Fella <nicolas.fella at gmx.de> wrote:

>
> On 2/4/26 10:16 AM, Ben Cooksley wrote:
>
> On Wed, Feb 4, 2026 at 10:05 PM Vlad Zahorodnii <vlad.zahorodnii at kde.org>
> wrote:
>
>> On 2/4/26 10:26 AM, Ben Cooksley wrote:
>>
>> On Wed, Feb 4, 2026 at 9:07 PM Ben Cooksley <bcooksley at kde.org> wrote:
>>
>>> On Wed, Feb 4, 2026 at 8:25 PM Vlad Zahorodnii <vlad.zahorodnii at kde.org>
>>> wrote:
>>>
>>>> Hello,
>>>>
>>>
>>> HI Vlad,
>>>
>>>
>>>>
>>>> Over approximately the past weekend something happened in our CI and
>>>> now
>>>> it takes quite long time for tests to run. For example, in kwin, we
>>>> have
>>>> a test that used to run for about 20 seconds, and now it takes about 5
>>>> or so minutes to finish running. Speaking for kwin, there were no
>>>> changes that could increase test run times so dramatically.
>>>>
>>>> January 26th:
>>>>
>>>>          Start  61: kwin-testOutputChanges
>>>>   61/158 Test  #61: kwin-testOutputChanges
>>>> .............................   Passed   19.36 sec
>>>>
>>>> January 29th:
>>>>
>>>>          Start  61: kwin-testOutputChanges
>>>>   61/158 Test  #61: kwin-testOutputChanges
>>>> .............................   Passed   43.93 sec
>>>>
>>>> January 30th:
>>>>
>>>>          Start  61: kwin-testOutputChanges
>>>>   61/158 Test  #61: kwin-testOutputChanges
>>>> .............................   Passed   45.91 sec
>>>>
>>>> Februrary 3rd:
>>>>
>>>>          Start  61: kwin-testOutputChanges
>>>>   61/158 Test  #61: kwin-testOutputChanges
>>>> .............................   Passed  254.19 sec
>>>>
>>>> FreeBSD appears to be fine.
>>>>
>>>> We suspect that test run times blew up due to enabling LSAN in various
>>>> libraries (kwin itself has no LSAN enabled yet). The issue doesn't
>>>> appear to be specific to only kwin, people reported that they've seen
>>>> similar issues in other projects too. Maybe something else happened to
>>>> CI that sysadmins will be able to clarify.
>>>>
>>>
>>> Nothing else happened to CI recently aside from the enablement of LSAN.
>>>
>>> The underlying SUSE images were for Qt 6.10 at least last rebuilt on
>>> January 25th, which is well before your "last good" date.
>>>
>>> The only change to CI between January 30th and February 3rd
>>> was fast_unwind_on_malloc=0 being added by default, even though it is
>>> primarily for the benefit of LSAN.
>>> I've now made changes to only set fast_unwind_on_malloc=0 if LSAN is
>>> explicitly enabled for a repository - hard to tell if that will fix the
>>> issue though as KWin takes a while to build.
>>>
>>
>> For the record, as per https://invent.kde.org/plasma/kwin/-/jobs/3958895
>> which completed moments ago:
>>
>> That branch contained some other things that could interfere with test
>> results. I started https://invent.kde.org/plasma/kwin/-/jobs/3959078 and
>> yeah it looks like test run times are back to the Jan 29-30th level.
>>
>> So, it seems like fast_unwind_on_malloc=0 is the culprit then?
>>
>
> Yes, quoting from https://source.android.com/docs/security/test/asan
>
> [quote]
> ASan uses a fast, frame-pointer-based unwinder to record a stack trace for
> every memory allocation and deallocation event in the program. Most of
> Android is built without frame pointers. As a result, you often get only
> one or two meaningful frames. To fix this, either rebuild the library with
> ASan (recommended!), or with:
>
> LOCAL_CFLAGS:=-fno-omit-frame-pointer
> LOCAL_ARM_MODE:=arm
>
> Or set ASAN_OPTIONS=fast_unwind_on_malloc=0 in the process environment.
> The latter can be very CPU-intensive, depending on the load.
> [/quote]
>
> That would align with what we're seeing here
>
> Hi,
>
> thanks for investigating this.
>
> I'm a bit surprised that fast_unwind_on_malloc seems to have an effect
> when LSAN is disabled. That said, we still should have the problem when
> LSAN *is* enabled, no?
>
> The reason I added fast_unwind_on_malloc=0 is because otherwise we would
> sometimes get unusable backtraces from LSAN, so I would not want to lose
> that.
>

You didn't lose it, see
https://invent.kde.org/sysadmin/ci-utilities/-/blob/master/components/EnvironmentHandler.py?ref_type=heads#L105
The fast_unwind_on_malloc flag will only be enabled though if LSAN is
enabled, which solves the problem for those who are badly affected by this.


> A possible remedy could be to build Qt with ASAN support. That's one
> additional flag in the buildsystem invokation. fast_unwind_on_malloc still
> would make sense to cover system libraries, but maybe it would be okay to
> do without that.
>

Building Qt is quite the endeavour, it isn't for the faint of heart and in
the case of WebEngine at least requires the VMLarge type on CI due to the
memory requirements involved (16GB isn't enough for WebEngine), and also
takes quite a long time.
That would significantly complicate building and updating VM images for CI
and make it significantly more resource intensive (not to mention the fact
that we would probably then have to build all the things that depend on Qt,
because we can't have the distribution install those anymore either).


> Cheers
>
> Nico
>
Thanks,
Ben
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.kde.org/pipermail/plasma-devel/attachments/20260205/4d2ee427/attachment.htm>