cstester - a tool for testing loading and painting of documents

[Boudewijn Rempt]

On Tuesday 18 January 2011, Thorsten Zachmann wrote:
> Hello all,
> 
> I'm currently working as part of my job for Nokia on cstester a tool to test 
> loading and painting for stage, tables and words. The aim of the tool is to 
> make sure there are no regressions added when features are modified. This is 
> done by loading the document and comparing some pre generated images of the 
> content with created images of the content. Also a good code coverage should 
> be possible to reach by using this method it uses a lot of code when doing 
> that.
> 
> The idea is to create a document set which contains features and test these 
> documents on a regularly basis.
> 
> There are still some open questions:
> - Most important one is how to make sure font rendering is always the same.
> One idea here is to create a virtual image that can be used to run the tests.

It should be enough that the daily build/test run machine stays substantially the same. We can use KO's cruncher1 for that, it already runs daily builds and unittests.

> - What is the best way to store the test set and images used for testing
> Should we create a new repository where the documents are stored.

Maybe a new subproject to calligra?

> - How to automatically run the tests to find regressions.

We can add it to cruncher1 -- it's a big, bad-ass multicore server that we (KO) have acquired especially for running calligra tests. benchmarks, unittests, stresstests...

> I will shortly commit the branch cstester-zachmann
> 
> How does it work
> 
> Creating the images used for verifying
> 
> cstester --create test.odp
> 
> will create the test images used for comparison. One needs to make sure the 
> images are as they are expected to be. The images to check will be created in 
> the directory test.odp.check so you need write access to the directory the file 
> is in.
> 
> Verifying that still all works fine
> 
> cstester --verify test.odp
> 
> This will load the document and create images of the content and then verify 
> them against the images saved in test.odp.check
> 
> It is possible to run the tool on multiple files at once by specifying muliple 
> files on the command line
> 
> cstester --create test1.odp test2.odp test3.ods test3.xls
> 
> Currently it has support for documents which are supported by stage and 
> tables. words is going to follow. Other document types should be easy to add.

Does it scroll through the entire document & also try the different slides and sheets?

> I also have plans for the future to extend the tool by:
> - adding a save / load circle to test that all is saved and loaded correctly 
> and we don't have any data loss there.
> - adding odf verification for the saved data

That's cool -- I think, though, that Jos van den Oever's test machine already implemented that, so we have scripts for that. The only problem is that I haven't had time to migrate that test server to calligra.

> 
> Also we are working inside Nokia on a set of documents that can be used for 
> this kind of testing. This documents will then be available for the community.
> 

Cool!
-- 
Boudewijn Rempt | http://www.valdyas.org, http://www.krita.org