QTestLib integration into ctest ? was [qt-dbus compilation problem]

[Ralf Habacker]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi all,

while porting qt-dbus to cmake, we encountered some problems with test
applications result display.

qt-dbus uses the QtTestLib framework for test applications, which
produces by default output in the following manner

C:\Daten\kde4\kdesupport\qt-dbus-build>bin\tst_hal
********* Start testing of tst_Hal *********
Config: Using QTest library 4.1.1, Qt 4.1.2
PASS   : tst_Hal::initTestCase()
<snip>
FAIL...
FAIL...
PASS   : tst_Hal::cleanupTestCase()
Totals: 2 passed, 2 failed, 0 skipped
********* Finished testing of tst_Hal *********

QtTestLib based test application has more options for example xml
output. See the complete list of options below.

C:\Daten\kde4\kdesupport\qt-dbus-build>bin\tst_hal -help
 Usage: bin\tst_hal [options] [testfunctions[:testdata]]...
    By default, all testfunction will be run.

 options:
 -functions : Returns a list of current testfunctions
 -xml       : Outputs results as XML document
 -lightxml  : Outputs results as stream of XML tags
 -o filename: Writes all output into a file
 -v1        : Print enter messages for each testfunction
 -v2        : Also print out each QVERIFY/QCOMPARE/QTEST
 -vs        : Print every signal emitted
 -eventdelay ms    : Set default delay for mouse and keyboard simulation
to ms milliseconds
 -keydelay ms      : Set default delay for keyboard simulation to ms
milliseconds
 -mousedelay ms    : Set default delay for mouse simulation to ms
milliseconds
 -keyevent-verbose : Turn on verbose messages for keyboard simulation
 -maxwarnings n    : Sets the maximum amount of messages to output.
                     0 means unlimited, default: 2000
 -help      : This help


ctest based test applications produces an output like shown below

Running tests...
Start processing tests
Test project
  1/  6 Testing tst_qdbusconnection           ***Failed
  2/  6 Testing tst_qdbusmarshall             ***Failed
  3/  6 Testing tst_qdbusxmlparser               Passed
  4/  6 Testing tst_qdbusinterface            ***Failed
  5/  6 Testing tst_qdbusabstractadaptor         Passed
  6/  6 Testing tst_hal                       ***Failed

The question to the cmake people is now how to integrate the QTestLib
based tests into ctest so that more informations are available as
mentioned by Thiago

>>> Would be better if CTest integrated with QtTestLib to find out
>>> *which* tests are available and, therefore, report the error
>>> count by test function, not test program.

Some topics:

- - output layout, may be similar to:

1/  6 Testing tst_qdbusconnection   5 tests passed, 10 tests ***Failed
2/  6 Testing tst_qdbusmarshall     8 tests passed  0 failed

or in a more detailed mode

  1/  6 Testing tst_qdbusconnection   6 tests
	PASS   : tst_Hal::initTestCase()
	FAIL!  : tst_Hal::getDevices() 'reply.type() =
	 QDBusMessage::ReplyMessage' returned FALSE. ()
	FAIL!  : tst_Hal::lock() Compared values are not the same
	Actual (spy.count): 0
	   Expected (3): 3
	PASS   : tst_Hal::cleanupTestCase()

  2/  6 Testing tst_qdbusmarshall     8 tests passed


- - How should cmake detect that the test application uses QtTestlib ?
- -> by a parameter for ENABLE_TESTING(qttestlib)

- - How should cmake get the number of available test functions ?
- -> by counting the number of lines provided by the -function option or
by  a specific command line switch which return the number of tests

Are there other proposals and suggestions ?

Additional requirements to the QtTestLib are welcome too as Thiago
Macieira mentioned

>>>If anything is required of QtTestLib, now is the time to ask too.

I think that such a support would affect whole KDE sources because there
are already QtTestLib based test in kdelibs and will be too in other
package.

Regards
 Ralf


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.0 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFEgGy4oHh+5t8EXncRAv7kAJ9XriXdSkEZSyzdyCzpFjM+ysnbcACfeHnD
X+7bXidC9HJ85p5gbL9mS4U=
=lXML
-----END PGP SIGNATURE-----