Automated Unit tests for the Future

[Alexander Neundorf]

Hi,

On Thursday 06 August 2009, Andrew Manson wrote:
> Hi everyone ,
>
> I'm a Marble developer and have just started to think seriously about unit
> tests and test driven development, possibly a bit late in the game but
> better late then never! I was surprised to see that we already had some
> unit tests in Marble but what was worse was that the core developers (
> including myself ) were "Surprised to see" that more than 50% of the tests
> failed. This is why I've come to you lot, hopefully to spark some
> discussion and get some things done ;)

Nice, I'd also like to get things going in this regard.

> The shocking thing about ^^^ is that the core developers were completely
> unaware that the unit tests were failing, which i find a bit odd when there
> are some pretty good Dashboard programs out there that can display and
> notify about the results of unit tests. I would really like to see one of
> these programs being used in a big way in KDE sooner rather than later so
> thats why I spend most of yesterday chatting with someone that knows much
> more about this stuff than I do. Sorry in advance if this turns out to be a
> very long email.
>
> Over the course of yesterday's discussions we have identified two major
> "styles" of unit test dashboard system. 1) a distributed model using
> something like CTest and CDash to display the results or 2) a centralised
> model where the build system is "part" of the *dedicated* unit test running
> system and the dashboard, like buildbot and CruiseControl . Each have their
> merits and hopefully we can discuss which one would be "more right" for
> KDE.
>
> For the centralised model we have a few organisational problems, mainly
> that we would need a (dedicated ) server to be provided so that we could
> run the build and display the results. This may not be so difficulty
> because I know that "Dirk" has a system like this setup already where the
> build results are displayed here: http://ktown.kde.org/~dirk/dashboard/ but
> this currently only works for displaying the actual result of the build and
> does not include unit tests. Using this model to incorporate unit tests
> shouldn't be too hard but it might cause an organisational nightmare for
> the sysadmins ( who have a hard enough time already ). On the other side of
> that, if we used a buildbot system and some of the new cool buzzwords in
> computer science like "distributed virtualised cloud computing" we could
> make a really cool system that would be able to check the build of KDE on
> Linux, Windows and Mac. This would be pretty cool but like i said an
> organisational nightmare.
>
> The other possibility is that we could use the distributed unit test
> reporting model that is currently incorporated by the CTest and CDash
> system. This is favorable for a number of reasons:
> 1) we are currently using CMake so adding CTest support is *easy*!
> 2) we don't have to have a centralised build system because any time the
> unit tests are run on any computer the results are submitted to the CDash
> dashboard 

Nevertheless having some big central machine ideally running a bunch of 
virtual machines, i.e. one with Windows, with OSX, FreeBSD and Solaris would 
be nice :-)
I guess this would mean an Apple server...

> 3) to set up the CDash system we would only need to add a few
> CMake variables to our CMakeList.txt files and we will be submitting
> results to a database in no time
> 4) from what I hear the kind people at http://www.cdash.org/CDash/index.php
> have already offered to host the CDash installation so our sysadmins would
> be able to take it easy.

Yes, Kitware offers free hosting for free projects at my.cdash.org.
I have already set up dashboards for kdelibs, kdepimlibs, kdesupport and 
automoc. I still need to document this somewhere, and I need help with 
running the builds :-)

> There are a lot of good points for using the CDash system but there is one
> pretty big problem with it that may render our test results somewhat
> useless. The fact that we are now starting to use the QtTest system makes
> things very easy for us but it means that each QtTest executable will be
> regarded as a single test by the CTest system. This is conceptually wrong
> because each QtTest executable contains many sub tests that can fail or
> pass independently of the single QtTest executable. Currently the CTest
> system only creates a test result on the test executable level which means
> that the results may not give any direct information as to why the test
> failed. Some may say that this is only a small detail and results on the
> executable level are "good enough" but if we are building a KDE-wide build
> system we should at least try and get it as close to perfect as we possibly
> can, or at least move in that direction. This problem could possibly be
> fixed with a patch to the CTest system but that would require some effort
> by someone far smarter than myself ;)

I don't think it would be too hard.
And you are not the first one with this issue. Not using ctest/cdash because 
of this one issue would be stupid, instead we (the people interested in 
testing) should contribute and write a patch :-)
I can commit to the cmake (ctest) cvs, so this wouldn't be a big problem. 
Not sure whether it would also require modifications in cdash. I don't have 
commit access there, but Julien, the main developer, is usually very 
responsive.

Alex

P.S. I'm offline now until Tuesday