QTestLib integration into ctest ? was [qt-dbus compilation problem]

[William A. Hoffman]

At 12:52 PM 6/2/2006, Ralf Habacker wrote:
>-----BEGIN PGP SIGNED MESSAGE-----
>Hash: SHA1
>
>Hi all,
>
>while porting qt-dbus to cmake, we encountered some problems with test
>applications result display.
>
>qt-dbus uses the QtTestLib framework for test applications, which
>produces by default output in the following manner
>
>C:\Daten\kde4\kdesupport\qt-dbus-build>bin\tst_hal
>********* Start testing of tst_Hal *********
>Config: Using QTest library 4.1.1, Qt 4.1.2
>PASS   : tst_Hal::initTestCase()
><snip>
>FAIL...
>FAIL...
>PASS   : tst_Hal::cleanupTestCase()
>Totals: 2 passed, 2 failed, 0 skipped
>********* Finished testing of tst_Hal *********
>
>QtTestLib based test application has more options for example xml
>output. See the complete list of options below.
>
>C:\Daten\kde4\kdesupport\qt-dbus-build>bin\tst_hal -help
> Usage: bin\tst_hal [options] [testfunctions[:testdata]]...
>    By default, all testfunction will be run.
>
> options:
> -functions : Returns a list of current testfunctions
> -xml       : Outputs results as XML document
> -lightxml  : Outputs results as stream of XML tags
> -o filename: Writes all output into a file
> -v1        : Print enter messages for each testfunction
> -v2        : Also print out each QVERIFY/QCOMPARE/QTEST
> -vs        : Print every signal emitted
> -eventdelay ms    : Set default delay for mouse and keyboard simulation
>to ms milliseconds
> -keydelay ms      : Set default delay for keyboard simulation to ms
>milliseconds
> -mousedelay ms    : Set default delay for mouse simulation to ms
>milliseconds
> -keyevent-verbose : Turn on verbose messages for keyboard simulation
> -maxwarnings n    : Sets the maximum amount of messages to output.
>                     0 means unlimited, default: 2000
> -help      : This help
>
>
>ctest based test applications produces an output like shown below
>
>Running tests...
>Start processing tests
>Test project
>  1/  6 Testing tst_qdbusconnection           ***Failed
>  2/  6 Testing tst_qdbusmarshall             ***Failed
>  3/  6 Testing tst_qdbusxmlparser               Passed
>  4/  6 Testing tst_qdbusinterface            ***Failed
>  5/  6 Testing tst_qdbusabstractadaptor         Passed
>  6/  6 Testing tst_hal                       ***Failed
>
>The question to the cmake people is now how to integrate the QTestLib
>based tests into ctest so that more informations are available as
>mentioned by Thiago
>
>>>> Would be better if CTest integrated with QtTestLib to find out
>>>> *which* tests are available and, therefore, report the error
>>>> count by test function, not test program.
>
>Some topics:
>
>- - output layout, may be similar to:
>
>1/  6 Testing tst_qdbusconnection   5 tests passed, 10 tests ***Failed
>2/  6 Testing tst_qdbusmarshall     8 tests passed  0 failed
>
>or in a more detailed mode
>
>  1/  6 Testing tst_qdbusconnection   6 tests
>        PASS   : tst_Hal::initTestCase()
>        FAIL!  : tst_Hal::getDevices() 'reply.type() =
>        QDBusMessage::ReplyMessage' returned FALSE. ()
>        FAIL!  : tst_Hal::lock() Compared values are not the same
>        Actual (spy.count): 0
>           Expected (3): 3
>        PASS   : tst_Hal::cleanupTestCase()
>
>  2/  6 Testing tst_qdbusmarshall     8 tests passed
>
>
>- - How should cmake detect that the test application uses QtTestlib ?
>- -> by a parameter for ENABLE_TESTING(qttestlib)
>
>- - How should cmake get the number of available test functions ?
>- -> by counting the number of lines provided by the -function option or
>by  a specific command line switch which return the number of tests
>
>Are there other proposals and suggestions ?
>
>Additional requirements to the QtTestLib are welcome too as Thiago
>Macieira mentioned
>
>>>>If anything is required of QtTestLib, now is the time to ask too.
>
>I think that such a support would affect whole KDE sources because there
>are already QtTestLib based test in kdelibs and will be too in other
>package.

You can always do ctest -V and get a more verbose output right now.
We could change ctest to maybe have some regular expressions that can
match output and produce output.

ADD_TEST(foo EXTRA_OUTPUT_MATCH ".*tests passed.* failed")

Then we could change QtTestLib to print out a line like this:
5 tests passed, 10 tests 

Then ctest would look for the extra output match in the tests output,
and print out something like this:

1/  6 Testing tst_qdbusconnection   5 tests passed, 10 tests ***Failed
2/  6 Testing tst_qdbusmarshall     8 tests passed  0 failed

I suppose the other option is to modify ctest to explicitly look for 
QtTestLib output.   I would like to try to come up with a solution that would
work for other projects as well.

-Bill