PIM Test Suite

[Ben Cooksley]

On Wed, Dec 12, 2018 at 6:37 AM Ben Cooksley <bcooksley at kde.org> wrote:
>
> On Tue, Dec 11, 2018 at 11:18 PM Daniel Vrátil <dvratil at kde.org> wrote:
> >
> > On Tuesday, 11 December 2018 09:22:20 CET Ben Cooksley wrote:
> > > Hi all,
> >
> > Hi Ben,
>
> Hi Daniel,

Hi Daniel,

>
> >
> > >
> > > Recently it seems something has changed in the PIM test suite, because
> > > i've started noticing that PIM projects are getting hung up waiting
> > > for kdeinit5 to exit.
> >
> > I'm not aware of any such change and we don't directly interact with kdeinit5
> > in our testsuite as far as I can tell. Can you link to such a test run? I
> > can't see any test on the CI that would be stuck waiting for kdeinit5.
>
> I'll grab a link next time I see it.
> (I tend to login to the builders, look for stuck builds and nudge them
> along, so don't always have the Jenkins links to hand)
>
> >
> > > Problem is, they should never, ever, ever, ever be running kdeinit5,
> > > because the CI test tooling does that for you. If you run it, then
> > > you'll end up stuck for eternity, because CTest will see the
> > > persistently running process and will wait forever (until someone
> > > intervenes manually and kills it off).
> > >
> > > Can the code that's killing the kdeinit process launched by the CI
> > > tooling please be removed?
> > >
> > > On that note, the hanging akonadiserver issue still hasn't been resolved.
> >
> > Checked all recent builds that run the Akonadi server as part of a test, but
> > haven't seen any of them stuck (or being manually terminated). But we lost all
> > the build history recently, so I don't have many logs to check :(
>
> You'll need to look for tests that take an extreme amount of time
> (like 18 hours) as I tend to just kill the hung process, which let's
> it continue on and finish normally.
>
> >
> > Is there a way I could get the Docker image (or at least the Dockerfile) that
> > the builds run in on the CI so I can try to reproduce this in an environment
> > that is as close to the CI env as possible? It doesn't seem to happen when I
> > just run the Akonadi testsuite inside my regular session - and I haven't
> > noticed kdeinit5 getting killed after running Akonadi tests locally, so I
> > suspects there's something else in the play here.
>
> The Docker images used by the CI system can be found on Dockerhub, in
> the kdeorg/ namespace.
>
> >
> > Anyway, if there's no easy way for me to set up the CI-like env locally, then
> > probably the best way is to try to investigate this directly on the CI builder
> > once it happens again, at least to see what state the whole thing is in.
> >
> > Btw, is there maybe a way to set a maximum run time for a build job on Jenkins
> > after which Jenkins simply kills the job? This would avoid us blocking the
> > builders, at least until this is fixed properly.
>
> Unfortunately not. We try to enforce this through CTest, but
> unfortunately this is a CTest behavioural quirk which means that if
> the test is finished but it's waiting for hung processes then it will
> wait for eternity.

I've just observed a job showing this behaviour tonight.

In this case, it was one of the more unusual cases whereby the whole
akonadicontrol, akonadiserver and all the agents were still running.
Usually some of these processes are no longer in the picture (often,
it's the knut ones hanging around still).

Log is here: https://build.kde.org/job/Applications/job/kdepim-addons/job/kf5-qt5%20SUSEQt5.10/12/consoleText

More curiously though, as a matter of chance the first process I
killed off was mysql.
The moment I did that, all of the Akonadi processes followed suit and
exited by themselves.

That isn't the usual behaviour - in many cases I have to go and kill
off each process in turn (often I tend to start with akonadicontrol,
which might explain why I haven't seen this before though)

Cheers,
Ben

>
> >
> > Cheers,
> > Dan
> >
>
> Cheers,
> Ben
>
> > >
> > > Thanks,
> > > Ben
> >
> >
> > --
> > Daniel Vrátil
> > www.dvratil.cz | dvratil at kde.org
> > IRC: dvratil on Freenode (#kde, #kontact, #akonadi, #fedora-kde)
> >
> > GPG Key: 0x4D69557AECB13683
> > Fingerprint: 0ABD FA55 A4E6 BEA9 9A83 EA97 4D69 557A ECB1 3683