Skyrocketing test run times

[Ben Cooksley]

On Wed, Feb 4, 2026 at 8:25 PM Vlad Zahorodnii <vlad.zahorodnii at kde.org>
wrote:

> Hello,
>

HI Vlad,


>
> Over approximately the past weekend something happened in our CI and now
> it takes quite long time for tests to run. For example, in kwin, we have
> a test that used to run for about 20 seconds, and now it takes about 5
> or so minutes to finish running. Speaking for kwin, there were no
> changes that could increase test run times so dramatically.
>
> January 26th:
>
>          Start  61: kwin-testOutputChanges
>   61/158 Test  #61: kwin-testOutputChanges
> .............................   Passed   19.36 sec
>
> January 29th:
>
>          Start  61: kwin-testOutputChanges
>   61/158 Test  #61: kwin-testOutputChanges
> .............................   Passed   43.93 sec
>
> January 30th:
>
>          Start  61: kwin-testOutputChanges
>   61/158 Test  #61: kwin-testOutputChanges
> .............................   Passed   45.91 sec
>
> Februrary 3rd:
>
>          Start  61: kwin-testOutputChanges
>   61/158 Test  #61: kwin-testOutputChanges
> .............................   Passed  254.19 sec
>
> FreeBSD appears to be fine.
>
> We suspect that test run times blew up due to enabling LSAN in various
> libraries (kwin itself has no LSAN enabled yet). The issue doesn't
> appear to be specific to only kwin, people reported that they've seen
> similar issues in other projects too. Maybe something else happened to
> CI that sysadmins will be able to clarify.
>

Nothing else happened to CI recently aside from the enablement of LSAN.

The underlying SUSE images were for Qt 6.10 at least last rebuilt on
January 25th, which is well before your "last good" date.

The only change to CI between January 30th and February 3rd
was fast_unwind_on_malloc=0 being added by default, even though it is
primarily for the benefit of LSAN.
I've now made changes to only set fast_unwind_on_malloc=0 if LSAN is
explicitly enabled for a repository - hard to tell if that will fix the
issue though as KWin takes a while to build.


>
> Either way, the current state of CI is not great. Hypothetically, test
> timeouts can be increased but QSignalSpy's have hardcoded timeouts that
> can be too low for the current CI. And in case of kwin, 5 minutes for a
> test is simply not a workable thing.
>
> Regards,
> Vlad
>

Thanks,
Ben
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.kde.org/pipermail/plasma-devel/attachments/20260204/ade7b489/attachment.htm>