Notice of intention to remove tests from KCrash and KNotifications

[Harald Sitter]

On Wed, Nov 6, 2019 at 8:07 AM Ben Cooksley <bcooksley at kde.org> wrote:
>
> On Tue, Nov 5, 2019 at 11:50 PM Harald Sitter <sitter at kde.org> wrote:
> >
> > I get where you are coming from, but honestly, none of this makes
> > pushing unreviewed commits that disable the entire test collection of
> > an entire framework any more correct. If it had only affected the one
> > test I wouldn't mind so much, but since it hasn't it only goes to
> > proof that we have reviews for a reason. In particular for repos that
> > are part of frameworks where we rely on tests to tell us if the
> > frameworks are good for the monthly release.
>
> Unfortunately reviews require responsive developers.
>
> In this instance, both Frameworks have been found to have developers
> which are not responsive (because my previous requests have been
> ignored), so sending a patch for review would have been pointless -
> because there would have been nobody to review it.
>
> Therefore bypassing review and taking any necessary action against the
> offending code (even if that does happen to be all of the tests) to
> correct the situation is the only viable option forward.

This is really not true.
We got the workaround for KCrash landed in not even half a business
day - including review.

> > There are many shades of grey between sending a "someone please fix"
> > mail to a mailing list and the nuclear option you implemented. The
> > most notable one being that you can ask someone who worked on the
> > repo, or tests recently "Hey, X, can you please disable the test on
> > windows because its impairing CI?" to which the answer is probably yes
> > because you'd not only address a very specific person but also the
> > task is doable in less than 5 minutes.
> > I understand that there's an element of cat herding in this, but
> > quality must matter for our products, and the quality of frameworks is
> > very tightly linked to autotesting and reviews because of how we
> > release them.
>
> This would require spending more time on the issue, which is something
> i'd very much rather avoid.
>
> It is expensive enough having to login to the CI builders to clear out
> these jobs when they get stuck (looking at anything that uses Akonadi
> especially here, along with these two Frameworks) and then asking the
> project lists to please fix their faulty code (which is then ignored,
> indicating that the concept of community maintained does not really
> work).
>
> Having to then lookup someone and then forward it on to them requires
> yet more time, with no guarantee that it will work - especially
> because some contributors are still hostile to any platform that is
> not FOSS and outright refuse to do anything to help those platforms.
>
> In the event that it does not work - then what would I do? Pick on the
> next person in the list (requiring yet more time)?
>
> Sorry, but that is simply too expensive, and if that is the policy
> you'd like to run with, then i'll stick to stripping projects of their
> test execution privileges outright across all platforms in the event
> they cause issues like this (which if ECM is anything to go by, is
> something people won't even notice even if the commit that puts that
> in effect is CC'ed to the project mailing list, which means that
> nobody will notice when the tests break because the CI system won't be
> running them)

I did not express myself well enough.
What I'd like you to do is talk to someone, not anyone, a very
specific someone of your choosing, to take care of the issue for you.
IOW: The sysadmins should delegate to a developer if the sysadmins
find themselves unable to accurately deal with a code problem (such as
a failing test that needs deactivating). If you tell an actually
relevant developer that's obviously better, if you don't have time to
look at the git log then that's fine too.

There are two different work items to differentiate here:

1. fixing something
2. disabling something because a fix was not made in an acceptable time frame

1 likely requires domain-knowledge of the specific framework and
probably quit a bit of time.
2 would generally only require understanding of cmake and a fairly
small time investment.

Pretty much any dev can do task 2. You just need to make someone care,
and the easiest way is to talk to a buddy directly and have them have
a look. Like if you ask me to deal with this for sysadmins I'll
definitely do. So, you have my permission to ask me to disable tests
on your behalf ;)

In short the process I would like is:

1. mail to list "yo, this framework has broken test xyz that is
severely impairing CI. plz fix"
2. if nothing happens ask someone (e.g. me) to force an action on the
repo (i.e. disable a specific test)
3. if the someone starts ghosting you you can still go in and do what
you did the other day

If you do 2 that has little overhead and a good chance of netting you
less work, not more.

HS