<div dir="ltr"><div dir="ltr">On Thu, Feb 5, 2026 at 12:14 AM Nicolas Fella <<a href="mailto:nicolas.fella@gmx.de">nicolas.fella@gmx.de</a>> wrote:</div><div class="gmail_quote gmail_quote_container"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><u></u>
<div>
<p><br>
</p>
<div>On 2/4/26 10:16 AM, Ben Cooksley wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">On Wed, Feb 4, 2026 at 10:05 PM Vlad Zahorodnii
<<a href="mailto:vlad.zahorodnii@kde.org" target="_blank">vlad.zahorodnii@kde.org</a>>
wrote:</div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div>On 2/4/26 10:26 AM, Ben Cooksley wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">On Wed, Feb 4, 2026 at 9:07 PM Ben
Cooksley <<a href="mailto:bcooksley@kde.org" target="_blank">bcooksley@kde.org</a>>
wrote:</div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div dir="ltr">On Wed, Feb 4, 2026 at 8:25 PM
Vlad Zahorodnii <<a href="mailto:vlad.zahorodnii@kde.org" target="_blank">vlad.zahorodnii@kde.org</a>>
wrote:</div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hello,<br>
</blockquote>
<div><br>
</div>
<div>HI Vlad,</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
Over approximately the past weekend
something happened in our CI and now <br>
it takes quite long time for tests to run.
For example, in kwin, we have <br>
a test that used to run for about 20
seconds, and now it takes about 5 <br>
or so minutes to finish running. Speaking
for kwin, there were no <br>
changes that could increase test run times
so dramatically.<br>
<br>
January 26th:<br>
<br>
Start 61: kwin-testOutputChanges<br>
61/158 Test #61: kwin-testOutputChanges <br>
............................. Passed
19.36 sec<br>
<br>
January 29th:<br>
<br>
Start 61: kwin-testOutputChanges<br>
61/158 Test #61: kwin-testOutputChanges <br>
............................. Passed
43.93 sec<br>
<br>
January 30th:<br>
<br>
Start 61: kwin-testOutputChanges<br>
61/158 Test #61: kwin-testOutputChanges <br>
............................. Passed
45.91 sec<br>
<br>
Februrary 3rd:<br>
<br>
Start 61: kwin-testOutputChanges<br>
61/158 Test #61: kwin-testOutputChanges <br>
............................. Passed
254.19 sec<br>
<br>
FreeBSD appears to be fine.<br>
<br>
We suspect that test run times blew up due
to enabling LSAN in various <br>
libraries (kwin itself has no LSAN enabled
yet). The issue doesn't <br>
appear to be specific to only kwin, people
reported that they've seen <br>
similar issues in other projects too. Maybe
something else happened to <br>
CI that sysadmins will be able to clarify.<br>
</blockquote>
<div><br>
</div>
<div>Nothing else happened to CI recently
aside from the enablement of LSAN.</div>
<div><br>
</div>
<div>The underlying SUSE images were for Qt
6.10 at least last rebuilt on January 25th,
which is well before your "last good" date.</div>
<div><br>
</div>
<div>The only change to CI between January
30th and February 3rd
was fast_unwind_on_malloc=0 being added by
default, even though it is primarily for the
benefit of LSAN.</div>
<div>I've now made changes to only
set fast_unwind_on_malloc=0 if LSAN is
explicitly enabled for a repository - hard
to tell if that will fix the issue though as
KWin takes a while to build.</div>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>For the record, as per <a href="https://invent.kde.org/plasma/kwin/-/jobs/3958895" target="_blank">https://invent.kde.org/plasma/kwin/-/jobs/3958895</a>
which completed moments ago:</div>
</div>
</div>
</blockquote>
<p>That branch contained some other things that could
interfere with test results. I started <a href="https://invent.kde.org/plasma/kwin/-/jobs/3959078" target="_blank">https://invent.kde.org/plasma/kwin/-/jobs/3959078</a>
and yeah it looks like test run times are back to the
Jan 29-30th level. </p>
<p>So, it seems like fast_unwind_on_malloc=0 is the
culprit then?</p>
</div>
</blockquote>
<div><br>
</div>
<div>Yes, quoting from <a href="https://source.android.com/docs/security/test/asan" target="_blank">https://source.android.com/docs/security/test/asan</a></div>
<div><br>
</div>
<div>[quote]</div>
ASan uses a fast, frame-pointer-based unwinder to record a
stack trace for every memory allocation and deallocation event
in the program. Most of Android is built without frame
pointers. As a result, you often get only one or two
meaningful frames. To fix this, either rebuild the library
with ASan (recommended!), or with:<br>
<br>
LOCAL_CFLAGS:=-fno-omit-frame-pointer<br>
LOCAL_ARM_MODE:=arm</div>
<div class="gmail_quote"><br>
<div>Or set ASAN_OPTIONS=fast_unwind_on_malloc=0 in the
process environment. The latter can be very CPU-intensive,
depending on the load. </div>
<div>[/quote]</div>
<div><br>
</div>
<div>That would align with what we're seeing here</div>
</div>
</div>
</blockquote>
<p>Hi,</p>
<p>thanks for investigating this.</p>
<p>I'm a bit surprised that fast_unwind_on_malloc seems to have an
effect when LSAN is disabled. That said, we still should have the
problem when LSAN *is* enabled, no?<br>
<br>
</p>
<p>The reason I added fast_unwind_on_malloc=0 is because otherwise
we would sometimes get unusable backtraces from LSAN, so I would
not want to lose that.</p></div></blockquote><div><br></div><div>You didn't lose it, see <a href="https://invent.kde.org/sysadmin/ci-utilities/-/blob/master/components/EnvironmentHandler.py?ref_type=heads#L105">https://invent.kde.org/sysadmin/ci-utilities/-/blob/master/components/EnvironmentHandler.py?ref_type=heads#L105</a></div><div>The fast_unwind_on_malloc flag will only be enabled though if LSAN is enabled, which solves the problem for those who are badly affected by this.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>
<p>A possible remedy could be to build Qt with ASAN support. That's
one additional flag in the buildsystem invokation. fast_unwind_on_malloc
still would make sense to cover system libraries, but maybe it
would be okay to do without that.</p></div></blockquote><div><br></div><div>Building Qt is quite the endeavour, it isn't for the faint of heart and in the case of WebEngine at least requires the VMLarge type on CI due to the memory requirements involved (16GB isn't enough for WebEngine), and also takes quite a long time.</div><div>That would significantly complicate building and updating VM images for CI and make it significantly more resource intensive (not to mention the fact that we would probably then have to build all the things that depend on Qt, because we can't have the distribution install those anymore either).</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>
<p>Cheers</p>
<p>Nico</p></div></blockquote><div>Thanks,</div><div>Ben </div></div></div>