Skyrocketing test run times

[Vlad Zahorodnii]

Hello,

Over approximately the past weekend something happened in our CI and now 
it takes quite long time for tests to run. For example, in kwin, we have 
a test that used to run for about 20 seconds, and now it takes about 5 
or so minutes to finish running. Speaking for kwin, there were no 
changes that could increase test run times so dramatically.

January 26th:

         Start  61: kwin-testOutputChanges
  61/158 Test  #61: kwin-testOutputChanges 
.............................   Passed   19.36 sec

January 29th:

         Start  61: kwin-testOutputChanges
  61/158 Test  #61: kwin-testOutputChanges 
.............................   Passed   43.93 sec

January 30th:

         Start  61: kwin-testOutputChanges
  61/158 Test  #61: kwin-testOutputChanges 
.............................   Passed   45.91 sec

Februrary 3rd:

         Start  61: kwin-testOutputChanges
  61/158 Test  #61: kwin-testOutputChanges 
.............................   Passed  254.19 sec

FreeBSD appears to be fine.

We suspect that test run times blew up due to enabling LSAN in various 
libraries (kwin itself has no LSAN enabled yet). The issue doesn't 
appear to be specific to only kwin, people reported that they've seen 
similar issues in other projects too. Maybe something else happened to 
CI that sysadmins will be able to clarify.

Either way, the current state of CI is not great. Hypothetically, test 
timeouts can be increased but QSignalSpy's have hardcoded timeouts that 
can be too low for the current CI. And in case of kwin, 5 minutes for a 
test is simply not a workable thing.

Regards,
Vlad