Kdevelop4 gsoc

[Andreas Pakulat]

On 06.05.08 21:41:12, Manuel Breugelmans wrote:
> 1. How will the different test(cases) be discovered by the framework?

What exactly does this mean? So far I only know QtTest, JUnit and Squish
as testing frameworks and the first one do "discovering" by using a
special base class, the latter by having a config file describing the
testcase scripts. Running of tests is always done by executing a binary
(eventually with some special parameters).

> a/ let this be the responsability of the user, ie require him/her to write a 
> seperate piece of code that collects all the tests
> b/ re-use the framework specific registration method, eg for CppUnit this is 
> basicly built-in
> c/ present selectable list of known classes, deduce testcases with the help of 
> duchain?
> 
> The first option is obviously very flexible but not user friendly. The second 
> is easy for certain frameworks eg it's builtin in CppUnit, however QTest has 
> no such facilities. The last seems do-able as well, it will need some 
> adaption for the C framework, Check (no classes).

Well, b+c should both be done, eventually moving c) to the end of the
Soc, depending on your plans and time. Wether a) is needed mostly
depends on the answer to the above and wether there are cases where you
want full control over "discovering testcases".

> 2. How to compile the tests, what kind of user input is required here?
> 
> a/ just let the user wite his own build rules with system of choice
> b/ automate it for cmake based projects? request only include dir's and 
> libraries/objects to link to.
> 
> Option a is the easiest implementation wise but not so user friendly. If this 
> route is taken 1. is a non-issue ... Maybe support this first and extend to 
> b/ later.

Both are needed IMHO. a) can be easy in larger projects such as KDE4
which already have unit-tests and thus its basically just a one-liner to
add a new test. However if you're not in such a project it would be nice
to use the buildtools. Using the Buildsystem API you can even do it
easily for all buildsystems that kdevelop supports, because you can just
tell the buildsystem to add a new target item with a couple of sources
attached.

One thing I just thought of: Are different testing-tools/API's going to
be different plugins or do you want to encapsulate all in one plugin?
IMHO the former is better, using common widgets and stuff available
from a library (maybe even more than just that), because it allows to
more easily add support later for other testing tools - specifically
using scripting languages. I guess I should really try to look into the
code, but my time is really limited right now (hopefully not anymore so
in a week or two).

> 3. How to determine the entry points and actually execute the tests?

I'm not sure what you mean with entry points.

> a/ run seperate executables for the different tests, not optimal at all ... 
> slow
> b/ use dynamic linking to inject the freshly compiled test objects eg with 
> QLibrary/QPluginLoader
> 
> Option b is preferable for sure, easy for CppUnit. Much faster both while 
> compiling and linking + when actually executing. Trouble with QtTest since it 
> only supports building executable(s). Might get funny with PyUnit ... maybe 
> it's better to wait for Kross support?

Executing tests should IMHO be done the same way its done without
kdevelop, we just want to make it easy to do so - i.e. right-click a
testcase in the GUI and say "execute", eventually even doing that for
just one of the slots in a QtTest (the executable test does support
this). Then we should display the results in a nice way.

I don't see people using KDevelop to run all the tests in larger
projects - thats something that nightly test runs should do. However I
think people will want to run those tests that test the components they
just changed.

Also I'm not sure what you gain from linking all testcases into one
executable, except for the case where you want to run all/nearly all of
the tests.

> 4. How to fetch the test results and pipe them into qxrunner?
> 
> this depends on the route taken for 3, obviously.
> 
> a/ if using seperate executables, produce xml results and stream parse it back 
> in, which will be slow (eg with QXmlStreamReader)

Thats not slow, parsing XML with QXmlStreamReader is quite fast
actually, especially as test-result information usually isn't overly
complex.

> b/ if using dlsym/open and friends this is easy just use some TestResult-like 
> class directly
> 
> When looking at these the main problem is QtTest's limited interface. All the 
> internals are private, the only real link is qExec + the convention that 
> commands are private slots. To bypass this either (i) a slow hacky route 
> needs to be taken (compile into executable(s) -> run seperatly -> parse XML 
> results),

Do you have numbers for this to be slow? I don't think it'll be that
slow, unless you have large projects and execute all tests, which
doesn't happen that often during development - IMHO.

> or (ii) duplicate (ie fork) QTestLib and make the internals public, 

No, definetly no. Thats not an Option.

> (iii) motivate Trolltech to expose more of the internals, not going to
> happen.

Not before Qt5, as that would most probably be binary incompatible.

Andreas

-- 
Green light in A.M. for new projects.  Red light in P.M. for traffic tickets.