[sysadmin/ci-tooling] build-specs/Plasma: Disable execution of tests for plasma-integration.

[Ben Cooksley]

On Sat, Jan 26, 2019 at 10:35 AM Albert Astals Cid <aacid at kde.org> wrote:
>
> El dilluns, 21 de gener de 2019, a les 6:46:32 CET, Ben Cooksley va escriure:
> > Git commit f6c79ff4787148459aa91c17d683e4fd6a57c323 by Ben Cooksley.
> > Committed on 21/01/2019 at 05:46.
> > Pushed by bcooksley into branch 'master'.
> >
> > Disable execution of tests for plasma-integration.
> > This is necessary to ensure CI nodes do not become blocked due to hanging tests withing plasma-integration.
> >
> > Currently, plasma-integration has several tests that make use of KIO slaves directly (skipping KLauncher).
> > Unfortunately, they do not terminate the slaves prior to the conclusion of the test, resulting in the kioslave processes being left around afterwards.
> > This is a condition that CTest will not tolerate, leading to it waiting indefinitely for these processes to exit - and in turn blocking all other builds on the CI node in question.
> > While this is not a major issue in the case of Linux builds, it can quickly become a severe condition in the case of FreeBSD and Windows builds due to those builders being fixed rather than dynamically allocated.
> >
> > This class of issue (CTest waiting due to resident processes being left behind) has been a major issue as of late and is quickly leading to the CI system becoming unmaintainable due to the level of breakage.
> > Should it be necessary to ensure the maintainability of the system, withdrawal of execution of tests for all projects is an option currently under consideration.
>
> Stopping running the tests is never the solution.
>
> We need to fix the tests.

That would be the ideal solution yes. Unfortunately in most cases I
don't get a response from the developers involved with the code, at
least not without some followup.
Having to login to the various Linux container / FreeBSD VM / Windows
VM machines every day to kill off hung processes isn't sustainable
though.

>
> And to fix the tests we need to be able to reproduce failures.

To my knowledge, with the exception of the Akonadi hangs, all of the
other tests were either reproducible by the developers or I haven't
heard back from them regarding the failures.

>
> Is this error only happening on the CI? If so is there an easy way someone can reproduce the setup that happens in our CI?
>
> If not we should be working on that.

For Linux builds I can provide instructions for how to reproduce the
CI environment - the majority of it comes from a Docker image so it
should be fairly easy for people to get up and running.

You'll want a decent internet connection though as you'll need to
download the KDE binaries built by the CI system, which are refreshed
as part of the Dependency Builds each week and as the system performs
builds - so these aren't something you can download once and keep
reusing. To provide some context Frameworks alone represents about
1.7GB - and with applications having dependencies on other parts of
KDE you'll need to download additional binaries on top of this.

For FreeBSD it's somewhat more complicated as you'd need to setup a
virtual machine that matches the CI environment (Tobias and Adriaan
look after this so they'd know more about the specifics required
here).

Windows is similar in needing you to setup a virtual machine matching
the CI environment. Due to licensing restrictions we can't provide a
disk image for Windows.

>
> I understand your frustration when CI breaks, but I have seem that often the answer for such issues is "it works for me and i can't reproduce the CI problem".
>
> So if we had a simple "run docker with this parameter and then run these 3 scripts to reproduce the error" answer I'm sure more people would try to fix them. At least i might.

A few more steps than 3 is required, but for the most part that is the case yes.

>
> Cheers,
>   Albert

Cheers,
Ben

>
> >
> > CCMAIL: plasma-devel at kde.org
> > CCMAIL: kde-frameworks-devel at kde.org
> > CCMAIL: release-team at kde.org
> > CCMAIL: kdevelop-devel at kde.org
> > CCMAIL: sysadmin at kde.org
> >
> > M  +2    -0    build-specs/Plasma/plasma-integration.yaml
> >
> > https://invent.kde.org/sysadmin/ci-tooling/commit/f6c79ff4787148459aa91c17d683e4fd6a57c323
> >
> > diff --git a/build-specs/Plasma/plasma-integration.yaml b/build-specs/Plasma/plasma-integration.yaml
> > index 3d39455..159546c 100644
> > --- a/build-specs/Plasma/plasma-integration.yaml
> > +++ b/build-specs/Plasma/plasma-integration.yaml
> > @@ -1,5 +1,7 @@
> >  kf5-qt5:
> > +  run-tests: False
> >    force-inject-asan: True
> >
> >  stable-kf5-qt5:
> > +  run-tests: False
> >    force-inject-asan: True
> >
>
>
>
>