Thoughts about a better Quality Management process for Plasma

Thu Jan 17 11:22:04 UTC 2013

On Sun, Jan 13, 2013 at 2:50 PM, Martin Gräßlin <mgraesslin at kde.org> wrote:
> Hi all,
>
> *warning* long mail! Please take time reading it. Given how long I am now
> writing on it: schedule half an hour or so ;-)
>
> I will try to formulate some thoughts on what I think might help to improve
> the Quality Management in Plasma. Some ideas are based on what works well in
> KWin.
>
> First of all I want to say that I find it awesome how many mentioned Quality
> in Aaron's 4.10 reflection thread. It means that we identified an area for
> improvement. I think that's the most important part to actually get something
> changed. And considering where we have been just a few years ago, I find it
> even more awesome that we consider Quality as the issue of the release cycle.
> We are complaining here on a very high scale. Yes we had regressions, but we
> also fixed them in time for the release.
>
> I think the problem with our QM process is, that we don't have a tool to
> support it. Our bugtracker is (in it's current state) totally useless. Let me
> just show a few stats for Plasma:
> * 858 bugs (including wishlist) created since 4.9.0 tagging
> ** 343 are marked as resolved
> *** 43 (!) are fixed - that are 5 % of all reported bugs
> *** 29 are invalid
> *** 222 are duplicates - a quote of 26 %
> ** 384 are still unconfirmed (45 %)
> *** 96 of those are crash reports
> ** 130 of the not resolved bugs are still in component general (25 % of the
> not resolved bugs)
>
> now for comparison I show the stats of KWin:
> * 432 bugs (including wishlist) created since 4.9.0 tagging
> ** 329  are marked as resolved
> *** 87 are fixed - that are 20 % of all reported bugs
> *** 28 are invalid
> *** 149 are duplicates - a quote of 34 %
> ** 42 are still unconfirmed (9.7 %)
> *** 4 of those are crash reports
> ** 17 of the not resolved (or needsinfo) bugs are still in component general
>
> Two people work on the KWin bugs. Plasma gets twice the amount of bug reports
> than KWin, so it would need 4 people to keep the bug tracker in a managable
> state. Looking at the stats it would need 4 people triaging one bug per day to
> keep it clean. That is doable!
>
> I have two requirements for bug reports:
> 1. There should not be any unconfirmed bugs
> 2. There should not be any bugs in component general
>
> What is an unconfirmed bug? It's a bug which has not been reproduced/verified
> that it is valid! A bug report should not stay in this status for a long time.
> Our users spend time to report it, so we should process them. If the user
> provided the correct input, it should be possible to get the bug to status
> "CONFIRMED". If the user cannot provide the necessary data to get the bug into
> status CONFIRMED, it has to be set to RESOLVED WAITINGFORINFO. A bug should
> never be longer than one week in status UNCONFIRMED without any action. If the
> user provided input which is not sufficient: ask for more. If he cannot
> provide it -> WAITINGFORINFO. Yes it sounds harsh, but you want to keep the
> tracker clean.
>
> What is a bug in component general? A bug that applies to everything of
> Plasma? Impossible! There is no general. A bug in general means that it is
> unknown where it belongs to. It should be moved into the best fitting
> component. If it doesn't exist create a new component! But don't keep it in
> general - nobody is responsible for those bugs.
>
This morning I spent an hour going through the list of "general", I
found a few problems.

There were several bugs I looked at where the relevant component
didn't exist (webslice plasmoid, timer plasmoid).
All plasmoids should have a component, it's much easier to manage lots
of small lists than one big one, especially for delegating. I created
components for the items I found (hope that's OK), and will continue
to do so. If anyone else finds one of these, ping someone with
bugzilla superpowers (like me)

There are a lot of crashes filed against general, looking at the
backtrace it's impossible to see who's at fault. The backtrace only
shows some timer firing, or a python binding call or something generic
- it's impossible to tell what's at fault. I have no idea how we can
fix this.

Some bugs weren't even against plasma (as devs know it). As we use the
phrase "plasma workspaces" I guess it's natural for bugs to end up
here which are totally generic. I found bugs on kwin, dolphin, and
other KDE things. I don't think there's really a fix for this.

There's a lot of wishlists.
I found these very hard to triage because I can't say on someone
else's project saying "we're never doing this, wontfix", which I would
do on my projects (maybe slightly more politely). I imagine many
others are in the same position, how can we manage this?

> Now the action items I would suggest to do:
>
> *Re-evaluate all components for Plasma*
> Let's check whether the components we have are still valid and fitting. Let's
> rename what can be improved and create what makes sense to create. Maybe even
> rethink whether it makes sense to split the product "plasma" into multiple
> products.
>
> *Find a maintainer for each component*
> The current maintainer model of Aaron and Marco for everything doesn't scale.
> We have so many people who could take over maintainership for a small
> component. Whoever is maintainer of a component, should become the default
> assignee for bugs in that component. No longer a one address for all assignee.
> Let's make the people responsible for their components.
>
> *Assign some "super" maintainers*
> Having a maintainer for each component carries the risk of people dropping out
> without being noticed and that would render the component unmaintained
> (example: all the kde-workspace components currently maintained by Lubos). We
> need some people who regularly check whether there is activity and poke the
> right people when lots of bugs get unmanaged (hey Jimmy are you still working
> on those? No? Should we get a new maintainer?). In Bugzilla talk this is
> called a QA contact IIRC. Random thought: give those super maintainers the
> power to pull the plug on a component if the quality decreases.
>
> *Restrict the available version numbers*
> At KWin we only allow bugs to be reported for:
> * the latest beta/RC
> * the last two minor versions of the current stable
> * the last two minor version of the last stable
> That is currently: 4.8.4, 4.8.5, 4.9.4, 4.9.5, 4.10 RC2
>
> We won't do a release for 4.8 or 4.9 any more, so we hardly care about these
> versions. There are just too many changes to properly support those versions.
> I would restrict even further, but there are users out there and it's not nice
> telling them that the distro they installed two months ago is already so
> outdated that we don't support it. The restriction on the latest two minor
> versions is to catch all the .1 bug reports we get through distro install
> media, but are fixed in .2 or .3. It helps the user to tell them: look there
> are bug fix releases, you might want to update.
>
> *Close all existing bug reports*
> We are currently in the process of developing Plasma 2 and porting everything
> to QML. Do we really care about a bug reported for the C++ version? Do we
> really care about a crash report which is for an old version and the backtrace
> doesn't match the code any more? Do we really want to spend person years of
> going through the bugs to confirm them? Now with the step towards Qt 5/KF
> 5/libplasma2/QML we have the chance to do the clear cut. We did it before
> (KDesktop/Kicker). Let's do it again. I'm pretty sure our users prefer the
> honesty that we won't work on it then having the bug open for so long.
> I just checked and there are 811 reports open which haven't changed over the
> last 365 days (and all of them are wishlist). And there are an additional 561
> bugs which have not changed over the last half year (no comment, nothing).
>
> That's all for the bugzilla part, now let's do some more general ideas I have
> :-)
>
> *Every commit should be referenced to a bug*
> What is the motivation for a commit? It's either a bug fix or it is a new
> feature/improvement. If it's a bug it's clear that there has to be a bug
> report for it (out of experience: there is always a bug, if not: create it).
> If it's a feature, it should also have a feature request in the tracker.
> Create it yourself if you need to. Sounds beaurocratic, but it comes quite
> handy as it allows to generate changelogs from it, allows people to easily
> test new features.
>
> *Bug fixes should come with a unit test*
> Plasma so far does not have any unit tests. Why? Let's not fool ourself with
> "it's UI, that's difficult to test". No it's not, because Plasma has a great
> separation between UI and logic. It would not be a problem to create a unit
> test for the data engines. If there is a bug do yourself the favor of creating
> the test. It helps you understanding the problem and fixing it. Also remember:
> we have now a Jenkins installation running the tests after each commit, so we
> see when they break (currently 3 tests in kde-workspace are failing).
>
> *New features should come with unit tests*
> If you work on something new it's easy to restructure the code in a way that
> it gets testable. New logic should be covered by tests. GUI is difficult,
> though we could spend a GSoC on getting a test framework.
>
> So that's it. Looking forwards to your comments on it :-)
>
> Cheers
> Martin
>
> _______________________________________________
> Plasma-devel mailing list
> Plasma-devel at kde.org
> https://mail.kde.org/mailman/listinfo/plasma-devel
>