Kdevelop4 gsoc

[Aleix]

On 5/10/08, Andreas Pakulat <apaku at gmx.de> wrote:
> On 06.05.08 21:41:12, Manuel Breugelmans wrote:
>  > 1. How will the different test(cases) be discovered by the framework?
>
>
> What exactly does this mean? So far I only know QtTest, JUnit and Squish
>  as testing frameworks and the first one do "discovering" by using a
>  special base class, the latter by having a config file describing the
>  testcase scripts. Running of tests is always done by executing a binary
>  (eventually with some special parameters).
>
>
>  > a/ let this be the responsability of the user, ie require him/her to write a
>  > seperate piece of code that collects all the tests
>  > b/ re-use the framework specific registration method, eg for CppUnit this is
>  > basicly built-in
>  > c/ present selectable list of known classes, deduce testcases with the help of
>  > duchain?
>  >
>  > The first option is obviously very flexible but not user friendly. The second
>  > is easy for certain frameworks eg it's builtin in CppUnit, however QTest has
>  > no such facilities. The last seems do-able as well, it will need some
>  > adaption for the C framework, Check (no classes).
>
>
> Well, b+c should both be done, eventually moving c) to the end of the
>  Soc, depending on your plans and time. Wether a) is needed mostly
>  depends on the answer to the above and wether there are cases where you
>  want full control over "discovering testcases".
Well, maybe we can provide the test case executables from the buildtool.

CMake can do that (because add_test() is used) so if you want, I can
expose it easily. QMake will do it easy also probably, I don't really
know about others.

>
>
>  > 2. How to compile the tests, what kind of user input is required here?
>  >
>  > a/ just let the user wite his own build rules with system of choice
>  > b/ automate it for cmake based projects? request only include dir's and
>  > libraries/objects to link to.
>  >
>  > Option a is the easiest implementation wise but not so user friendly. If this
>  > route is taken 1. is a non-issue ... Maybe support this first and extend to
>  > b/ later.
>
>
> Both are needed IMHO. a) can be easy in larger projects such as KDE4
>  which already have unit-tests and thus its basically just a one-liner to
>  add a new test. However if you're not in such a project it would be nice
>  to use the buildtools. Using the Buildsystem API you can even do it
>  easily for all buildsystems that kdevelop supports, because you can just
>  tell the buildsystem to add a new target item with a couple of sources
>  attached.
Well, you already know how I feel about editing user scripts, I don't
really think that messing with it is a good idea...

But anyway, my point is that you shouldn't do it buildtool dependent,
you should decide what you need and use the API and anything needed
will have to be implemented on the other side (maybe later on).

>
>  One thing I just thought of: Are different testing-tools/API's going to
>  be different plugins or do you want to encapsulate all in one plugin?
>  IMHO the former is better, using common widgets and stuff available
>  from a library (maybe even more than just that), because it allows to
>  more easily add support later for other testing tools - specifically
>  using scripting languages. I guess I should really try to look into the
>  code, but my time is really limited right now (hopefully not anymore so
>  in a week or two).
>
>
>  > 3. How to determine the entry points and actually execute the tests?
>
>
> I'm not sure what you mean with entry points.
>
>
>  > a/ run seperate executables for the different tests, not optimal at all ...
>  > slow
>  > b/ use dynamic linking to inject the freshly compiled test objects eg with
>  > QLibrary/QPluginLoader
>  >
>  > Option b is preferable for sure, easy for CppUnit. Much faster both while
>  > compiling and linking + when actually executing. Trouble with QtTest since it
>  > only supports building executable(s). Might get funny with PyUnit ... maybe
>  > it's better to wait for Kross support?
>
>
> Executing tests should IMHO be done the same way its done without
>  kdevelop, we just want to make it easy to do so - i.e. right-click a
>  testcase in the GUI and say "execute", eventually even doing that for
>  just one of the slots in a QtTest (the executable test does support
>  this). Then we should display the results in a nice way.
>
>  I don't see people using KDevelop to run all the tests in larger
>  projects - thats something that nightly test runs should do. However I
>  think people will want to run those tests that test the components they
>  just changed.
>
>  Also I'm not sure what you gain from linking all testcases into one
>  executable, except for the case where you want to run all/nearly all of
>  the tests.

I think we don't have to force the user to compile the tests in any
strange way, KDevelop just would have to understand which one is the
one that the user wants to run in each case.

>
>
>  > 4. How to fetch the test results and pipe them into qxrunner?
>  >
>  > this depends on the route taken for 3, obviously.
>  >
>  > a/ if using seperate executables, produce xml results and stream parse it back
>  > in, which will be slow (eg with QXmlStreamReader)
>
>
> Thats not slow, parsing XML with QXmlStreamReader is quite fast
>  actually, especially as test-result information usually isn't overly
>  complex.

I don't think XML is complex as well, and anyway, my point is that
first it has to work, then we will think how to make it work perfecly
(and fast).

>
>
>  > b/ if using dlsym/open and friends this is easy just use some TestResult-like
>  > class directly
>  >
>  > When looking at these the main problem is QtTest's limited interface. All the
>  > internals are private, the only real link is qExec + the convention that
>  > commands are private slots. To bypass this either (i) a slow hacky route
>  > needs to be taken (compile into executable(s) -> run seperatly -> parse XML
>  > results),
>
>
> Do you have numbers for this to be slow? I don't think it'll be that
>  slow, unless you have large projects and execute all tests, which
>  doesn't happen that often during development - IMHO.
>

This fork+execve path is very optimised on *nix systems, I don't see
what is the problem here.

>
>  > or (ii) duplicate (ie fork) QTestLib and make the internals public,
>
>
> No, definetly no. Thats not an Option.
>
>
>  > (iii) motivate Trolltech to expose more of the internals, not going to
>  > happen.
>
>
> Not before Qt5, as that would most probably be binary incompatible.
>

Why don't just let KDevelop adapt to the environment? We have good
tools around, just benefit from them all in a productive way!

Aleix

>  Andreas
>
>
>  --
>  Green light in A.M. for new projects.  Red light in P.M. for traffic tickets.
>
>
>  _______________________________________________
>  KDevelop-devel mailing list
>  KDevelop-devel at kdevelop.org
>  https://barney.cs.uni-potsdam.de/mailman/listinfo/kdevelop-devel
>