Test Detection [was: Re: Kdevelop4 gsoc]

[Andreas Pakulat]

On 11.05.08 14:29:33, Manuel Breugelmans wrote:
> On Saturday 10 May 2008 22:15:47 Andreas Pakulat wrote:
> > On 06.05.08 21:41:12, Manuel Breugelmans wrote:
> > > 1. How will the different test(cases) be discovered by the framework?
> >
> > What exactly does this mean? So far I only know QtTest, JUnit and Squish
> > as testing frameworks and the first one do "discovering" by using a
> > special base class, the latter by having a config file describing the
> > testcase scripts. Running of tests is always done by executing a binary
> > (eventually with some special parameters).
> 
> QtTests just inherit QObject, there is no root testcase. The only clue that 
> you are dealing with a testcase is a call to QtTest::qExec freestanding 
> function which takes a QObject pointer. Typically wrapped in a main-macro 
> such as QTEST_MAIN, KDEV_TESTMAIN etc. I'm not sure whether Kdevelop can peek 
> inside macros? Or optionally just their arguments, these are typically not 
> numerous so could be made configurable.

AFAIK the C++ support does allow to resolve macro's, but I don't know
wether thats "exported" to the outside world via DUChain. So the problem
is that you don't know which targets from the project model are
testcases right? See below for the rest.

> > > a/ let this be the responsability of the user, ie require him/her to
> > > write a seperate piece of code that collects all the tests
> > > b/ re-use the framework specific registration method, eg for CppUnit this
> > > is basicly built-in
> > > c/ present selectable list of known classes, deduce testcases with the
> > > help of duchain?
> > >
> > > The first option is obviously very flexible but not user friendly. The
> > > second is easy for certain frameworks eg it's builtin in CppUnit, however
> > > QTest has no such facilities. The last seems do-able as well, it will
> > > need some adaption for the C framework, Check (no classes).
> >
> > Well, b+c should both be done, eventually moving c) to the end of the
> > Soc, depending on your plans and time. Wether a) is needed mostly
> > depends on the answer to the above and wether there are cases where you
> > want full control over "discovering testcases".
> 
> In retrospect maybe a) should just be left out.

Reading the above, I agree.

> Strong testcase detection in 
> the framework specific plugins combined with some graphical selection method 
> should be enough.

Exactly. We can just put that information into our project model and the
user can get a nice context menu entry to tell Kdevelop: "Hey, this
target in the project is actually building a testcase". Or we can find
other GUI's for selecting which targets are testcases that should be
run and which testing framework they use.

> The main goal is to make test running fun and graphically attractive
> (alike the junit eclipse runner which does a great job.)

+1, I'll try to have a closer look at eclipse junit support next week at
work as I only had short looks so far.

Andreas

-- 
In the stairway of life, you'd best take the elevator.