Notice of intention to remove tests from KCrash and KNotifications

[Ben Cooksley]

On Thu, Nov 7, 2019 at 1:18 AM Harald Sitter <sitter at kde.org> wrote:
>
> On Wed, Nov 6, 2019 at 8:07 AM Ben Cooksley <bcooksley at kde.org> wrote:
> >
> > On Tue, Nov 5, 2019 at 11:50 PM Harald Sitter <sitter at kde.org> wrote:
> > >
> > > I get where you are coming from, but honestly, none of this makes
> > > pushing unreviewed commits that disable the entire test collection of
> > > an entire framework any more correct. If it had only affected the one
> > > test I wouldn't mind so much, but since it hasn't it only goes to
> > > proof that we have reviews for a reason. In particular for repos that
> > > are part of frameworks where we rely on tests to tell us if the
> > > frameworks are good for the monthly release.
> >
> > Unfortunately reviews require responsive developers.
> >
> > In this instance, both Frameworks have been found to have developers
> > which are not responsive (because my previous requests have been
> > ignored), so sending a patch for review would have been pointless -
> > because there would have been nobody to review it.
> >
> > Therefore bypassing review and taking any necessary action against the
> > offending code (even if that does happen to be all of the tests) to
> > correct the situation is the only viable option forward.
>
> This is really not true.
> We got the workaround for KCrash landed in not even half a business
> day - including review.
>

Unfortunately it is certainly my perception - which is based off of my
emails on the subject not receiving any response.
This experience is one that hasn't only happened with Frameworks - it
has also happened (several times) with PIM.

Taking direct action against a project is pretty much guaranteed to
trigger a fairly rapid response (which is exactly what happened in
this case).

It should be noted that saying i'm going to take action usually
results in either no response, or a response along the lines of "no,
you can't do that" rather than the issue actually being addressed
(which once again, is what happened in this case too)


> > > There are many shades of grey between sending a "someone please fix"
> > > mail to a mailing list and the nuclear option you implemented. The
> > > most notable one being that you can ask someone who worked on the
> > > repo, or tests recently "Hey, X, can you please disable the test on
> > > windows because its impairing CI?" to which the answer is probably yes
> > > because you'd not only address a very specific person but also the
> > > task is doable in less than 5 minutes.
> > > I understand that there's an element of cat herding in this, but
> > > quality must matter for our products, and the quality of frameworks is
> > > very tightly linked to autotesting and reviews because of how we
> > > release them.
> >
> > This would require spending more time on the issue, which is something
> > i'd very much rather avoid.
> >
> > It is expensive enough having to login to the CI builders to clear out
> > these jobs when they get stuck (looking at anything that uses Akonadi
> > especially here, along with these two Frameworks) and then asking the
> > project lists to please fix their faulty code (which is then ignored,
> > indicating that the concept of community maintained does not really
> > work).
> >
> > Having to then lookup someone and then forward it on to them requires
> > yet more time, with no guarantee that it will work - especially
> > because some contributors are still hostile to any platform that is
> > not FOSS and outright refuse to do anything to help those platforms.
> >
> > In the event that it does not work - then what would I do? Pick on the
> > next person in the list (requiring yet more time)?
> >
> > Sorry, but that is simply too expensive, and if that is the policy
> > you'd like to run with, then i'll stick to stripping projects of their
> > test execution privileges outright across all platforms in the event
> > they cause issues like this (which if ECM is anything to go by, is
> > something people won't even notice even if the commit that puts that
> > in effect is CC'ed to the project mailing list, which means that
> > nobody will notice when the tests break because the CI system won't be
> > running them)
>
> I did not express myself well enough.
> What I'd like you to do is talk to someone, not anyone, a very
> specific someone of your choosing, to take care of the issue for you.
> IOW: The sysadmins should delegate to a developer if the sysadmins
> find themselves unable to accurately deal with a code problem (such as
> a failing test that needs deactivating). If you tell an actually
> relevant developer that's obviously better, if you don't have time to
> look at the git log then that's fine too.
>
> There are two different work items to differentiate here:
>
> 1. fixing something
> 2. disabling something because a fix was not made in an acceptable time frame
>
> 1 likely requires domain-knowledge of the specific framework and
> probably quit a bit of time.
> 2 would generally only require understanding of cmake and a fairly
> small time investment.
>
> Pretty much any dev can do task 2. You just need to make someone care,
> and the easiest way is to talk to a buddy directly and have them have
> a look. Like if you ask me to deal with this for sysadmins I'll
> definitely do. So, you have my permission to ask me to disable tests
> on your behalf ;)
>
> In short the process I would like is:
>
> 1. mail to list "yo, this framework has broken test xyz that is
> severely impairing CI. plz fix"
> 2. if nothing happens ask someone (e.g. me) to force an action on the
> repo (i.e. disable a specific test)
> 3. if the someone starts ghosting you you can still go in and do what
> you did the other day
>
> If you do 2 that has little overhead and a good chance of netting you
> less work, not more.

That could potentially work (I will note though that it is basically
just shifting the problem - the problem being #1 not resulting in a
response).
I do have some concerns about the long term scalability of the approach though.

>
> HS

Cheers,
Ben