How to write UI tests (for sublime/ and shell/)

[Alexander Dymo]

As I've promised before, I sat down today and wrote a small howto on UI  
testing. This basically summarizes my experience writing tests for sublime/ 
and shell. Hope you'll find this useful. If something is missing, please ping 
me ;)

Same howto is on our wiki, if you have changes or fixes, feel free to modify: 
http://www.kdevelop.org/mediawiki/index.php/How_To_Write_UI_Tests

==============================================

== How to compile and run tests ==
Just configure kdevplatform with 
  cmake -DKDE4_BUILD_TESTS=on
and you're all set.

To run all tests use
  make test
command from toplevel dir or from sublime/ and shell/ subdirs.

To run individual test just run its executable:
  cd sublime/tests
  ./sublime-viewactivationtest

To run individual test function just pass the name of the function as the 
parameter:
  cd sublime/tests
  ./sublime-viewactivationtest testActivationAfterViewRemoval

You can pass several test functions there. Also there are more options that 
tests accept (verbosity and output control, etc.). See 
http://doc.trolltech.com/4.4/qtestlib-manual.html#qtestlib-command-line-arguments 
for the reference or just run test with --help from the command line.




== UI Testing Examples ==

=== Simple Shell test from scratch ===

==== shellexampletest.h ====
<pre>
#include <QObject>
#include <tests/common/autotestshell.h>
using namespace KDevelop;

class ShellExampleTest: public QObject {
    Q_OBJECT
private slots:
    void init();
    void cleanup();
    void testSomething();
};
</pre>

==== shellexampletest.cpp ====
<pre>
#include "shellexampletest.h"
#include <tests/common/kdevtest.h>
#include <QtTest/QtTest>

void ShellExampleTest::init() {
    AutoTestShell::init();  // create instance of test shell extension
    KDevelop::Core::initialize(); // initialize kdevplatform
}
void ShellDocumentOperationTest::cleanup() {
}
void ShellDocumentOperationTest::testSomething() {
    // your test here
    // expect full blown kdevplatform-based application 
    // loaded and initialized by this moment (albeit without any plugins 
because we currently don't have them for test shell)
}
//THIS IS VERY IMPORTANT: use KDEVTEST_MAIN macro for ALL UI tests
KDEVTEST_MAIN(ShellExampleTest)
#include "shellexampletest.moc"
</pre>

==== CMakeLists.txt ====
<pre>
set( EXECUTABLE_OUTPUT_PATH ${CMAKE_CURRENT_BINARY_DIR} )
include_directories(${CMAKE_SOURCE_DIR} ${CMAKE_SOURCE_DIR}/interfaces 
${CMAKE_SOURCE_DIR}/shell)

set( shellexampletest_SRCS shellexampletest.cpp )
kde4_add_unit_test(shell-exampletest ${shellexampletest_SRCS})
target_link_libraries(shell-exampletest ${QT_QTTEST_LIBRARY} 
${KDE4_KDEUI_LIBS} kdevplatformshell )
</pre>

=== Simple Sublime test from scratch ===
I won't post an example here because Sublime tests are just usual tests (but 
also created with KDEVTEST_MAIN macro). Some Sublime tests launch simple 
application, some of them just create several objects and test their 
functionality. Take a look at existing Sublime tests because 99% of the time 
you'll just add new test functions to them. 




== UI Testing Techniques ==
Writing UI tests is actually very easy and fun to do. Here are some useful UI 
test writing techniques.

=== Test mostly user experience instead internals ===
The main idea behind UI tests we write for Sublime and Shell is to replicate 
user actions and verify results that user would get. This way of thinking 
greatly simplifies tests, reduces the time you spend writing the test and 
actually assures that UI is never broken for the user.

Some guidelines:
* call high-level public functions that are directly called by the user 
interface and check their results, for example
    IDocumentController *documentController = 
Core::self()->documentController();
    documentController->openDocumentFromText("");
    QCOMPARE(documentController->openDocuments().count(), 1);

* find QAction objects in the UI and trigger them manually, for example:
    QAction *splitAction = 
Core::self()->uiControllerInternal()->activeMainWindow()->actionCollection()->action("split_vertical");
    QVERIFY(splitAction);
    splitAction->trigger();
    QCOMPARE(doc->views().count(), 2);

* simulate events, for example:
    qApp->sendEvent(myView->widget(), new QFocusEvent(QEvent::FocusIn));
    QCOMPARE(mainWindow->activeView(), myView);

* try to check signals you emit, especially if the signal is the part of 
public API (lots of plugins will connect to the signals and will expect them 
to work reliably), for example:
    QSignalSpy spy(controller, SIGNAL(viewAdded(Sublime::View*)));
    area->addView(doc->createView());
    QCOMPARE(spy.count(), 1);



===  Aggregate test results before making comparisons and verifications ===
With multiple QVERIFY and QCOMPARE test functions may become messy.
It is sometimes a good idea to print some data into string and compare that 
string to the original data also encoded in the string.

For an example, see sublime/tests/areaoperationtest.cpp. We print the layout 
of mainwindow's central area to the string and compare the string with 
expected layout:
 checkAreaViewsDisplay(mainWindow, area, QString("\n\
 [ vertical splitter ]\n\
     [ vertical splitter ]\n\
         [ view2.1.1 view2.1.2 ]\n\
         [ view2.4.1 ]\n\
     [ horizontal splitter ]\n\
         [ view2.5.1 ]\n\
         [ view2.3.1 ]\n\
 "));
This check assures that after we show the particular area in some mainwindow 
we get several splitters created and those splitters contain some visible 
views inside.

One such check replaces 12-15 QCOMPARE()'s and QVERIFY()'s and makes test look 
easier to understand because it's immediately obvious for the reader that 
mainwindow should have vertical splitters with two splitters inside. Both of 
internal splitters should contain some views, etc, etc.



=== Write less tests and more test functions ===
Each test class is linked to the separate executable. Linking too many 
executables may be really slow. Also each Sublime test is a full-featured KDE 
application and takes some time to start. Each Shell test is not only a KDE 
application but a KDevPlatform application so it starts even slower. 

So the morale is, do not spawn too many test executables without the real 
need. It is always possible to run just several test functions from the test 
in case you don't want to wait for the whole test to pass:
 #cd sublime/tests
 #./sublime-controllertest --functions
 testDocumentDeletion()
 testAreaDeletion()
 testNamedAreas()
 #./sublime-controllertest testNamedAreas testAreaDeletion
 ...



=== Keep test code clean ===
As I've said before, the goal of writing UI tests is to specify certain 
behavior we expect to work. Of course that specification should be succinct 
and easily understandable for the reader. Please expect that your test will 
be read by other developers. When they break your code, they will start 
looking from the test, not from the code.



=== Feel good ;) ===
Once you write the test for your feature or bugfix, you can feel safe and 
warm ;) Your work will never be broken/lost/etc. because there's now an 
automated way to assure that.