Thoughts about a better Quality Management process for Plasma

Sun Jan 13 14:50:25 UTC 2013

Hi all,

*warning* long mail! Please take time reading it. Given how long I am now 
writing on it: schedule half an hour or so ;-)

I will try to formulate some thoughts on what I think might help to improve 
the Quality Management in Plasma. Some ideas are based on what works well in 
KWin.

First of all I want to say that I find it awesome how many mentioned Quality 
in Aaron's 4.10 reflection thread. It means that we identified an area for 
improvement. I think that's the most important part to actually get something 
changed. And considering where we have been just a few years ago, I find it 
even more awesome that we consider Quality as the issue of the release cycle. 
We are complaining here on a very high scale. Yes we had regressions, but we 
also fixed them in time for the release.

I think the problem with our QM process is, that we don't have a tool to 
support it. Our bugtracker is (in it's current state) totally useless. Let me 
just show a few stats for Plasma:
* 858 bugs (including wishlist) created since 4.9.0 tagging
** 343 are marked as resolved
*** 43 (!) are fixed - that are 5 % of all reported bugs
*** 29 are invalid
*** 222 are duplicates - a quote of 26 %
** 384 are still unconfirmed (45 %)
*** 96 of those are crash reports
** 130 of the not resolved bugs are still in component general (25 % of the 
not resolved bugs)

now for comparison I show the stats of KWin:
* 432 bugs (including wishlist) created since 4.9.0 tagging
** 329  are marked as resolved
*** 87 are fixed - that are 20 % of all reported bugs
*** 28 are invalid
*** 149 are duplicates - a quote of 34 %
** 42 are still unconfirmed (9.7 %)
*** 4 of those are crash reports
** 17 of the not resolved (or needsinfo) bugs are still in component general

Two people work on the KWin bugs. Plasma gets twice the amount of bug reports 
than KWin, so it would need 4 people to keep the bug tracker in a managable 
state. Looking at the stats it would need 4 people triaging one bug per day to 
keep it clean. That is doable!

I have two requirements for bug reports:
1. There should not be any unconfirmed bugs
2. There should not be any bugs in component general

What is an unconfirmed bug? It's a bug which has not been reproduced/verified 
that it is valid! A bug report should not stay in this status for a long time. 
Our users spend time to report it, so we should process them. If the user 
provided the correct input, it should be possible to get the bug to status 
"CONFIRMED". If the user cannot provide the necessary data to get the bug into 
status CONFIRMED, it has to be set to RESOLVED WAITINGFORINFO. A bug should 
never be longer than one week in status UNCONFIRMED without any action. If the 
user provided input which is not sufficient: ask for more. If he cannot 
provide it -> WAITINGFORINFO. Yes it sounds harsh, but you want to keep the 
tracker clean.

What is a bug in component general? A bug that applies to everything of 
Plasma? Impossible! There is no general. A bug in general means that it is 
unknown where it belongs to. It should be moved into the best fitting 
component. If it doesn't exist create a new component! But don't keep it in 
general - nobody is responsible for those bugs.

Now the action items I would suggest to do:

*Re-evaluate all components for Plasma*
Let's check whether the components we have are still valid and fitting. Let's 
rename what can be improved and create what makes sense to create. Maybe even 
rethink whether it makes sense to split the product "plasma" into multiple 
products.

*Find a maintainer for each component*
The current maintainer model of Aaron and Marco for everything doesn't scale. 
We have so many people who could take over maintainership for a small 
component. Whoever is maintainer of a component, should become the default 
assignee for bugs in that component. No longer a one address for all assignee. 
Let's make the people responsible for their components.

*Assign some "super" maintainers*
Having a maintainer for each component carries the risk of people dropping out 
without being noticed and that would render the component unmaintained 
(example: all the kde-workspace components currently maintained by Lubos). We 
need some people who regularly check whether there is activity and poke the 
right people when lots of bugs get unmanaged (hey Jimmy are you still working 
on those? No? Should we get a new maintainer?). In Bugzilla talk this is 
called a QA contact IIRC. Random thought: give those super maintainers the 
power to pull the plug on a component if the quality decreases.

*Restrict the available version numbers*
At KWin we only allow bugs to be reported for:
* the latest beta/RC
* the last two minor versions of the current stable
* the last two minor version of the last stable
That is currently: 4.8.4, 4.8.5, 4.9.4, 4.9.5, 4.10 RC2

We won't do a release for 4.8 or 4.9 any more, so we hardly care about these 
versions. There are just too many changes to properly support those versions. 
I would restrict even further, but there are users out there and it's not nice 
telling them that the distro they installed two months ago is already so 
outdated that we don't support it. The restriction on the latest two minor 
versions is to catch all the .1 bug reports we get through distro install 
media, but are fixed in .2 or .3. It helps the user to tell them: look there 
are bug fix releases, you might want to update.

*Close all existing bug reports*
We are currently in the process of developing Plasma 2 and porting everything 
to QML. Do we really care about a bug reported for the C++ version? Do we 
really care about a crash report which is for an old version and the backtrace 
doesn't match the code any more? Do we really want to spend person years of 
going through the bugs to confirm them? Now with the step towards Qt 5/KF 
5/libplasma2/QML we have the chance to do the clear cut. We did it before 
(KDesktop/Kicker). Let's do it again. I'm pretty sure our users prefer the 
honesty that we won't work on it then having the bug open for so long.
I just checked and there are 811 reports open which haven't changed over the 
last 365 days (and all of them are wishlist). And there are an additional 561 
bugs which have not changed over the last half year (no comment, nothing).

That's all for the bugzilla part, now let's do some more general ideas I have 
:-)

*Every commit should be referenced to a bug*
What is the motivation for a commit? It's either a bug fix or it is a new 
feature/improvement. If it's a bug it's clear that there has to be a bug 
report for it (out of experience: there is always a bug, if not: create it). 
If it's a feature, it should also have a feature request in the tracker. 
Create it yourself if you need to. Sounds beaurocratic, but it comes quite 
handy as it allows to generate changelogs from it, allows people to easily 
test new features.

*Bug fixes should come with a unit test*
Plasma so far does not have any unit tests. Why? Let's not fool ourself with 
"it's UI, that's difficult to test". No it's not, because Plasma has a great 
separation between UI and logic. It would not be a problem to create a unit 
test for the data engines. If there is a bug do yourself the favor of creating 
the test. It helps you understanding the problem and fixing it. Also remember: 
we have now a Jenkins installation running the tests after each commit, so we 
see when they break (currently 3 tests in kde-workspace are failing).

*New features should come with unit tests*
If you work on something new it's easy to restructure the code in a way that 
it gets testable. New logic should be covered by tests. GUI is difficult, 
though we could spend a GSoC on getting a test framework.

So that's it. Looking forwards to your comments on it :-)

Cheers
Martin
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part.
URL: <http://mail.kde.org/pipermail/plasma-devel/attachments/20130113/21237d0d/attachment.sig>