[Kde-pim] Unittest policy?

[Volker Krause]

On Sunday 27 September 2009 08:05:58 Andras Mantia wrote:
> Thomas McGuire wrote:
> > On Saturday 26 September 2009 17:45:55 you wrote:
> >> I decided to run the entire kdepimlibs test suite...
> >>
> >> This resulted in that after >20 hours I terminated the suite (almost all
> >>  time spent waiting for akonadi tests to timeout) .
> >>
> >> I'm wondering if there is a policy about tests in kdepim[libs]?
> >> Or if there isn't should there be?
> >
> > There should be. And the policy should say that the tests always have to
> > pass. Otherwise these tests aren't useful anymore to detect actual
> > regressions, if we have to always remember which tests pass and which
> > don't.
>
> I'd say, that if we have 600+ tests and running them takes 20 hours, the
> tests aren't useful anymore. Even if running them takes 10 minutes you
> cannot expect developers to run at every check-in.
>  I'd say, you should run the tests for the module (sub-directory) you are
> working on.

20 hours is a massive exaggeration, the dashboard says it takes about half an 
hour. Of course this is still too much to run it for every commit, it's ten 
times as long as the entire build takes, and that's even without icecream. 
Running just the relevant sub-set and relying on nightly runs for the rest 
helps a bit with that (and is perfectly fine IMHO). Parallel test execution 
could speed things up on multi-core machines, but ctest can't do that AFAIK. 
Shouldn't be too hard to come up with a small script for that though.

> > Normally the Akonadi tests do exactly that, they set up an environment
> > that includes a MySQL server and the Akonadi server running in an
> > confined environment. If that doesn't work for you, there's a bug
> > somewhere.
>
> Although in the (distant) past they worked for me, I have the same problem,
> it doesn't work now. Interesting is that running the test manually works.

The setup got broken again somehow, the problem with interference from 
kbuildsycoca is back, we had fixed that twice already. This seems to be the 
main reason for their slowness and random hangs/failures. 
If you look at the Akonadi tests at dashboard you will notice that the failing 
ones usually change every day. So, the problem is not that the actual tests 
fail, but the setup is unstable.

Apart from the Akonadi tests, which need a rather complicated setup (and try 
to take care of that), we have several other ones that need D-Bus and/or X 
running. That's fine when you run them in your normal session, but it can be 
a problem for automated runs.

regards
Volker
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part.
URL: <http://mail.kde.org/pipermail/kde-pim/attachments/20090927/7ee26f7e/attachment.sig>
-------------- next part --------------
_______________________________________________
KDE PIM mailing list kde-pim at kde.org
https://mail.kde.org/mailman/listinfo/kde-pim
KDE PIM home page at http://pim.kde.org/