[dot] Report of the KDE Quality Assurance Meeting

[Dot Stories]

URL: http://dot.kde.org/1134473541/

From: Frerich Raabe <raabe at kde.org>
Dept: meesa-do-not-know
Date: Tuesday 13/Dec/2005, @03:32

Report of the KDE Quality Assurance Meeting
===========================================

     On the weekend of December 10th & 11th, a small group of nine KDE
contributors met in Hamburg to work on quality assurance checks for
KDE's code base. This not only covered C++ sourcecode but also other
aspects such as checking the state of the API documentation, looking for
common errors in KDE's manuals and evaluating the usability of KDE
applications.  Read on for the full report.

    Group Photo from the KDE QA Meeting: Simon, Ellen, Harri, Harald,
Adriaan, Frerich, Reggie, Dirk and Ben
     Hosted by the kind folks of froglogic [http://www.froglogic.com],
saturday started with people arriving at the froglogic office and saying
hello to each other. As people arrived, it became clear that there was
some interest in looking to what degree froglogic Squish
[http://www.froglogic.com/squish/] would be useful for testing the user
interface of KDE applications, so a small Squish demonstration took
place. Also, the original project (the infamous English Breakfast
Network [http://www.englishbreakfastnetwork.org]) which started the
whole idea of having a QA meeting wasn't known to some people so a
laptop was attached to the projector and a short "Browsing around the
EBN" session was held. It was a nice opportunity to show off how the
results of (different) automatic checks could be presented in a
reasonably uniform manner.

     After having some food, Ellen Reitmayr of OpenUsability.org fame
was kind enough to educate the rest of us about usability.  The fact
that a whole bunch of factors [http://www.openusability.org] which
influence the usability of the user interface can easily be checked
automatically came as a relieving surprise (so the idea wasn't all that
crazy after all!). During her explanations, it became apparent that even
though many things are terribly difficult to check automatically it
would be a big achievement if we could have nightly checks for the low
hanging fruits. Getting them out of the way (and there are a lot of
them) would give the usability people more time to focus on the things
which actually require human intervention.

     The rest of the afternoon was spent on letting ideas for checks
bounce off each other, brainstorming about the feasibility of automatic
checking for such things (just imagine people occasionally moaning
something like "You can't possibly do this and that automatically!" or
"Such a tool would give so many false hits, it'd clutter the output and
make the valid reports about issues harder to spot!"). A little bit of
hacking was done as well (talking about and looking into the sources of
different C++ parsers) - in particular, KDevelop
[http://www.kdevelop.org] developer Harald Fernengel hacked up an
QTestLib-based test based on Dirk's binary compatibility checker
(unfortunately nobody seems to know about 'gmake bcheck') which checks
whether a given interface composed of a number of C++ classes is binary
compatible to another interface, which would be useful for KDE's
libraries (so that your KDE applications don't break after upgrading
KDE's libraries to a new minor release because of binary
incompatibility). In the evening, people had food together and then
visited Hamburg's central Christmas fair, having some of the infamous
German Glühwein together. :-)

     While Saturday was much more about talking and exchanging ideas, a
lot more work got done on Sunday. Reginald hacked up a little Squish
script to show off how Squish can be used to automatically check the
menus of different KDE applications for some of the usability points
Ellen mentioned the day before (the picture of Squish opening and
closing all menus available in an application - and the fact that one
script was used to test different applications - caused some "oh!" and
"ah!" and nodding among the viewers). Adriaan
[http://www.kde.nl/people/ade.html] (who runs the English Breakfast
Network [http://www.englishbreakfastnetwork.org]) was given a free
Squish license so that you should see the results of various usability
tests showing up on the web pages. Work was also done on porting the
QTestLib-based tests in KDE's libraries to the new QTestLib as stored in
the qt-copy module of KDE's repository. In general, Sunday's soundscape
was much influenced by the sound of people typing on their laptop
keyboards. The day finished with a dinner together, after which the
first persons left to venture back to the mundane world.

     A few noteworthy things we got out of this weekend:

    * Two days were not enough. KDE has no real centralised testing
      framework in place, so a lot of time was spent on discussing very
      basic problems.
    * Usability is a lot less diffuse a term than most people think. A
      lot of things which make up the usability of a program (as defined
      by the KDE HIG) can indeed be checked automatically. Having a
      Squish license for running Squish on the EBN so that we can start
      generating the reports is a very nice achievement.
    * There doesn't seem to be a useful and free static C++ analyser,
      which is a pity. Maybe using Roberto Raggi's C++ parser to do
      something like that would be worthwhile, but it is still a lot of
      work.
    * How to present whatever results we get out of any checks and
      integrating that information into the workflow of volunteers
      working on KDE is a tricky issue - having five-digit numbers of
      issues on the front page easily scares potentially helping hands
      away. This is indeed a social problem, and we could have spent the
      entire weekend on just evaluating different approaches at how to
      present rules appropriately.
    * Visiting the Reeperbahn in Hamburg on a Sunday night is quite
      boring.