PIM Test Suite

[Ben Cooksley]

On Wed, Dec 12, 2018 at 11:29 PM Ben Cooksley <bcooksley at kde.org> wrote:
>
> On Wed, Dec 12, 2018 at 6:37 AM Ben Cooksley <bcooksley at kde.org> wrote:
> >
> > On Tue, Dec 11, 2018 at 11:18 PM Daniel Vrátil <dvratil at kde.org> wrote:
> > >
> > > On Tuesday, 11 December 2018 09:22:20 CET Ben Cooksley wrote:
> > > > Hi all,
> > >
> > > Hi Ben,
> >
> > Hi Daniel,
>
> Hi Daniel,
>
> >
> > >
> > > >
> > > > Recently it seems something has changed in the PIM test suite, because
> > > > i've started noticing that PIM projects are getting hung up waiting
> > > > for kdeinit5 to exit.
> > >
> > > I'm not aware of any such change and we don't directly interact with kdeinit5
> > > in our testsuite as far as I can tell. Can you link to such a test run? I
> > > can't see any test on the CI that would be stuck waiting for kdeinit5.
> >
> > I'll grab a link next time I see it.
> > (I tend to login to the builders, look for stuck builds and nudge them
> > along, so don't always have the Jenkins links to hand)
> >
> > >
> > > > Problem is, they should never, ever, ever, ever be running kdeinit5,
> > > > because the CI test tooling does that for you. If you run it, then
> > > > you'll end up stuck for eternity, because CTest will see the
> > > > persistently running process and will wait forever (until someone
> > > > intervenes manually and kills it off).
> > > >
> > > > Can the code that's killing the kdeinit process launched by the CI
> > > > tooling please be removed?
> > > >
> > > > On that note, the hanging akonadiserver issue still hasn't been resolved.
> > >
> > > Checked all recent builds that run the Akonadi server as part of a test, but
> > > haven't seen any of them stuck (or being manually terminated). But we lost all
> > > the build history recently, so I don't have many logs to check :(
> >
> > You'll need to look for tests that take an extreme amount of time
> > (like 18 hours) as I tend to just kill the hung process, which let's
> > it continue on and finish normally.
> >
> > >
> > > Is there a way I could get the Docker image (or at least the Dockerfile) that
> > > the builds run in on the CI so I can try to reproduce this in an environment
> > > that is as close to the CI env as possible? It doesn't seem to happen when I
> > > just run the Akonadi testsuite inside my regular session - and I haven't
> > > noticed kdeinit5 getting killed after running Akonadi tests locally, so I
> > > suspects there's something else in the play here.
> >
> > The Docker images used by the CI system can be found on Dockerhub, in
> > the kdeorg/ namespace.
> >
> > >
> > > Anyway, if there's no easy way for me to set up the CI-like env locally, then
> > > probably the best way is to try to investigate this directly on the CI builder
> > > once it happens again, at least to see what state the whole thing is in.
> > >
> > > Btw, is there maybe a way to set a maximum run time for a build job on Jenkins
> > > after which Jenkins simply kills the job? This would avoid us blocking the
> > > builders, at least until this is fixed properly.
> >
> > Unfortunately not. We try to enforce this through CTest, but
> > unfortunately this is a CTest behavioural quirk which means that if
> > the test is finished but it's waiting for hung processes then it will
> > wait for eternity.
>
> I've just observed a job showing this behaviour tonight.
>
> In this case, it was one of the more unusual cases whereby the whole
> akonadicontrol, akonadiserver and all the agents were still running.
> Usually some of these processes are no longer in the picture (often,
> it's the knut ones hanging around still).
>
> Log is here: https://build.kde.org/job/Applications/job/kdepim-addons/job/kf5-qt5%20SUSEQt5.10/12/consoleText
>
> More curiously though, as a matter of chance the first process I
> killed off was mysql.
> The moment I did that, all of the Akonadi processes followed suit and
> exited by themselves.
>
> That isn't the usual behaviour - in many cases I have to go and kill
> off each process in turn (often I tend to start with akonadicontrol,
> which might explain why I haven't seen this before though)

Has anyone had a chance to look into this?

>
> Cheers,
> Ben

Thanks,
Ben

>
> >
> > >
> > > Cheers,
> > > Dan
> > >
> >
> > Cheers,
> > Ben
> >
> > > >
> > > > Thanks,
> > > > Ben
> > >
> > >
> > > --
> > > Daniel Vrátil
> > > www.dvratil.cz | dvratil at kde.org
> > > IRC: dvratil on Freenode (#kde, #kontact, #akonadi, #fedora-kde)
> > >
> > > GPG Key: 0x4D69557AECB13683
> > > Fingerprint: 0ABD FA55 A4E6 BEA9 9A83 EA97 4D69 557A ECB1 3683