Setting up a Quality Team within KDE

[Volker Krause]

On Monday 16 April 2012 08:02:51 Andras Mantia wrote:
> On Thursday, April 12, 2012 10:49:10 PM Alexander Neundorf wrote:
> > Yes, how good squish works for you depends on at least two things:
> We also use Squish, and it found bugs and regressions in our code.
> Still, there is a big problem with it: the test needs to be maintained
> constantly. If they are not, thing will quickly escalade to the wrong
> direction, ie. you will end up with test cases that just fail and are
> hard to adapt.
> The difference between maintaining unit tests and squish tests is that
> squish tests the code through the UI and it is very easy to change the
> UI in a way it breaks the tests, unless the test case is well set up
> (which means it is not just "record and save").

I don't think UI is necessarily easier to change than internal API, the key 
difference is that you'll get a compile error for the unit test, while you 
wont notice a broken Squish test immediately. Therefore it's IMHO essential to 
have continuous or at least nightly runs of the Squish tests, fixing them 
while you still know what you changed is usually quick and easy.

> > * how well you set up your squish scripts, object map, etc. If done ad
> > hoc,  you end up in an unmaintainable mess. If done carefully, it
> > works.
> 
> Exactly. :)
> 
> > * and of course, what you actually test. We don't really test that
> > dialogs  etc. still look the same, we use squish to drive our
> > software, see that it doesn't crash while doing that, that the
> > expected actions work properly, and the produced output data is
> > correct. We actually try to do things directly in many cases to stay
> > independent from GUI details, i.e. we call slots and check Qt
> > properties directly from the python scripts.
> 
> This is something we might not agree, as the point of squish is to test
> the app through the UI. If you test through slots, in most cases you
> might write unit tests as well.
> 
> I thing Squish tests would improve KDE's code, but this will need a
> dedicated team (which we want to set up, i know :) ). The point is, that
> Squish is not magic, you will need to think, design, write code, just
> not in C++, but in Python for example.

While a dedicated team can help to get this started, I think it has to be the 
developers job to keep the tests working when they break as a result of 
changing code (which is exactly how it works with unit tests right now). 
Otherwise there's the risk of ending up with rotting tests which are useless.

regards,
Volker
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 190 bytes
Desc: This is a digitally signed message part.
URL: <http://mail.kde.org/pipermail/kde-core-devel/attachments/20120416/cdc264f1/attachment.sig>