Unit Testing

Fri Apr 13 07:09:16 UTC 2012

2012/4/9 Milian Wolff <mail at milianw.de>

> Hey all,
>
> today I finally took some time to review / test the unittest branches a
> bit.
> It's coming along nicely but there are still a few things to do:
>
> Hey, thanks! I'm sorry I didn't see your mail before.

> # Test Population
>
> Right now, the user must manually click the "reload" button in the test
> view
> to trigger a test-find-job for the loaded projects. That is very
> unintuitive
> and must be improved. For CTest I think that the tests should be added
> right
> from the cmakemanager, making the ctestprovider superflous (instead, make
> the
> cmakemanager a testprovider).

I realize that, and I have already changed some behavior to work in this
way. I left the Reload button there for cases where the automatic fetching
is not reliable and for removing tests.

> The question there is: How to "remove"
> obsoleted/removed unit tests?
>
> I.e. if a user removes a kde4_add_unit_test or similar in a CMakeLists.txt
> -
> how can we know that and remove the related testsuite from the
> testcontroller?
>
Not only that, tests can also be changed (stuff renamed, cases added or
removed). So far there is no functionality to check and report this. Being
rather time-limited (again), I kept the short-term solution of the Reload
button. I think removing test suites or cases is a rare-enough action.

>
> For the PHPUnit provider I'd similarily make the phpplugin itself the
> plugin
> provider (obsoleting the additional plugin). Then, we can just report new
> tests whenever we find a class inheriting from the appropriate PHPUnit
> class.
> The big question here is again removal of oudated tests... And
> additionally,
> the found tests should be cached I think, so that we do not have to reparse
> the whole project just to find the unit tests. Ideas?
>
> # QTest != CTest
>
> I see that the current ctest provider runs "$testbinary -functions" to find
> the test functions of a test suite, but that is a QTest feature... With
> ctest,
> you can add anything as a test and I'm not sure whether the above is
> "OK"...
> Ideas?
>
Well, it tries to run the test as if it's made with QTest. If it is not, no
cases are reported (as CTest doesn't support naming individual test cases),
but the suite can still be run. I checked this with some tests from the
Veritas plugin.

>
> # API
>
> The TestResult API needs to be extended, I think with at least a
> TestCaseResult member. After all, there could be unit tests without
> distinguishable child "functions" (see above). Furthermore, if a test fails
> completely (e.g. the PHP one when you don't have PHPUnit installed), then
> we
> need a way to mark the whole suite as failed. Probably more needs to be
> done.
>
Agreed, thet's why I indroduced a new struct.

>
> I furthermore severly dislike ITestController::notifyTestRunFinished() -
> this
> should be removed together with the testRunFinished. Instead, create a
> TestJob
> class that
>
> a) holds the testresult (this should *not* be put into the testsuite!)
> b) has the proper signals when a test failed/finished etc. pp.
>
Hm. I have thought about this, it probably is the better API, so that more
than one result per suite can be stored if needed. However, it would need
to be a subclass of OutputJob, so I'm not sure where exactly to put it. The
util library?

>
> # UI & WorkFlow
>
> The current UI needs some love, first of all - proper icons. Ideas? I don't
> like the current choice but have no idea what else we could use...
>
Neither do I, I specifically included different icons for different levels
to see which ones look better. I still don't know, input here would be
valuable.

Additionally, I don't really like the actions in the toolbar and context
menu. Any idea how could running, jumping to source be accomplished better?

>
> Furthermore, it's pretty bad that when you run multiple test suites, only
> the
> results of the last one stay visible. Instead, the jobs output should
> somehow
> be concatenated. And one must be able to jump to failures somehow, similar
> to
> the navigation between compiler errors.
>
That's similar to what running "ctest" does. It is something I have in
plan, but I'm not sure how it should look yet.

>
> The tree view also does not indicate which suite is currently being run, at
> least some icon should be changed or so.
>
That's because I don't know which icon this should be. Those "task-*" icons
seem to be out of place there.

>
> Very important for me would also be a way to quickly run unit tests. I
> think
> we need some dialog, similar to what you see when you press "j" in Kmail to
> jump to a different folder. There, the last executed test suite should be
> pre-
> selected. Typing filters the result set and enter would execute all visible
> tests.
>
> # Conclusion
>
> I hope you all start testing the unittest branch and helping out. Many
> thanks
> to Miha for starting this, good job! Lets hope we can ship 4.4 with this
> feature!
>
Thank you for the feedback.

Unforturtunately, I must say that my schoolwork has caught up with me again
(you may notice the drop in my activity), and I think my progress will be a
little slow for the next couple of weeks as well. In any case, I'll do my
best.

If it's OK, I'll introduce a common TestJob, change the API for test
results, and/or complete the automatic finding of tests next.

>
> Bye
>
> --
> Milian Wolff
> mail at milianw.de
> http://milianw.de
> --
> KDevelop-devel mailing list
> KDevelop-devel at kdevelop.org
> https://barney.cs.uni-potsdam.de/mailman/listinfo/kdevelop-devel
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.kde.org/pipermail/kdevelop-devel/attachments/20120413/2c495bcb/attachment.html>