Unit Testing

[Milian Wolff]

Hey all,

today I finally took some time to review / test the unittest branches a bit. 
It's coming along nicely but there are still a few things to do:

# Test Population

Right now, the user must manually click the "reload" button in the test view 
to trigger a test-find-job for the loaded projects. That is very unintuitive 
and must be improved. For CTest I think that the tests should be added right 
from the cmakemanager, making the ctestprovider superflous (instead, make the 
cmakemanager a testprovider). The question there is: How to "remove" 
obsoleted/removed unit tests?

I.e. if a user removes a kde4_add_unit_test or similar in a CMakeLists.txt - 
how can we know that and remove the related testsuite from the testcontroller?

For the PHPUnit provider I'd similarily make the phpplugin itself the plugin 
provider (obsoleting the additional plugin). Then, we can just report new 
tests whenever we find a class inheriting from the appropriate PHPUnit class. 
The big question here is again removal of oudated tests... And additionally, 
the found tests should be cached I think, so that we do not have to reparse 
the whole project just to find the unit tests. Ideas?

# QTest != CTest

I see that the current ctest provider runs "$testbinary -functions" to find 
the test functions of a test suite, but that is a QTest feature... With ctest, 
you can add anything as a test and I'm not sure whether the above is "OK"... 
Ideas?

# API

The TestResult API needs to be extended, I think with at least a 
TestCaseResult member. After all, there could be unit tests without 
distinguishable child "functions" (see above). Furthermore, if a test fails 
completely (e.g. the PHP one when you don't have PHPUnit installed), then we 
need a way to mark the whole suite as failed. Probably more needs to be done.

I furthermore severly dislike ITestController::notifyTestRunFinished() - this 
should be removed together with the testRunFinished. Instead, create a TestJob 
class that 

a) holds the testresult (this should *not* be put into the testsuite!)
b) has the proper signals when a test failed/finished etc. pp.

# UI & WorkFlow

The current UI needs some love, first of all - proper icons. Ideas? I don't 
like the current choice but have no idea what else we could use...

Furthermore, it's pretty bad that when you run multiple test suites, only the 
results of the last one stay visible. Instead, the jobs output should somehow 
be concatenated. And one must be able to jump to failures somehow, similar to 
the navigation between compiler errors.

The tree view also does not indicate which suite is currently being run, at 
least some icon should be changed or so.

Very important for me would also be a way to quickly run unit tests. I think 
we need some dialog, similar to what you see when you press "j" in Kmail to 
jump to a different folder. There, the last executed test suite should be pre-
selected. Typing filters the result set and enter would execute all visible 
tests.

# Conclusion

I hope you all start testing the unittest branch and helping out. Many thanks 
to Miha for starting this, good job! Lets hope we can ship 4.4 with this 
feature!

Bye

-- 
Milian Wolff
mail at milianw.de
http://milianw.de
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part.
URL: <http://mail.kde.org/pipermail/kdevelop-devel/attachments/20120409/5feff73c/attachment.sig>