Tests hanging on CI

[Ben Cooksley]

On Mon, Jan 21, 2019 at 2:40 AM Friedrich W. H. Kossebau
<kossebau at kde.org> wrote:
>
> Am Donnerstag, 17. Januar 2019, 10:43:56 CET schrieb Ben Cooksley:
> > On Sun, Jan 13, 2019 at 9:27 PM Ben Cooksley <bcooksley at kde.org> wrote:
> > > Hi KDevelop Devs,
> >
> > Hi folks,
> >
> > > Currently we have an issue where the KDevelop's test_cmakemanager test
> > > doesn't cleanup after itself properly on the CI system. This leads to
> > > it leaving behind a "cmake -E server" process, which due to how CTest
> > > works means it will wait indefinitely.
> > >
> > > This requires manual intervention to sort out.
> > >
> > > Could someone please investigate and sort this out?
> > >
> > > A log showing the issue can be found at
> > > https://build.kde.org/job/KDevelop/job/kdevelop/job/kf5-qt5%20SUSEQt5.10/2
> > > 6/consoleFull
>
> The issues seems to be that the test crashes. Which should be fixed
> independently. But in any case, this can happen as it does now, so IMHO we
> should be prepared for that, and not have the whole build fail on that.

*nod*

>
> Any code in the test to shutdown the cmake server will not be effective if the
> test itself crashes, right?

That's correct, unless you were to do it in a crash handler (which is
probably unreliable as the test process has crashed already).
I know other projects in the past have written wrappers to go around
the test itself - and as long as the wrapper didn't crash, this should
allow for cleanup where needed.

>
> Is there no way to somehow log by the test execution environment what other
> processes a test triggers and collect them as last resort, if the test itself
> fails to do so? Or could any such processes be somehow explicitly registered
> somewhere. so the test environment can gracefully pick up the parts on a
> crashed test?

Unfortunately that would have to be done by CTest - which is the one
doing the blocking here :(

Our testing harness (the Python scripts) don't know where CTest is at
in terms of executing the tests, so their ability to intevene here
would be fairly limited - and i'm not sure how well a grandparent is
able to monitor children processes (I think it might be possible, how
cross platform that code would be though is another matter)

>
> Cheers
> Friedrich
>
>

Cheers,
Ben