Is there an accepted/official ODF test suite?

[Thorsten Zachmann]

Hello,

On Thursday, April 12, 2012 20:05:51 Friedrich W. H. Kossebau wrote:
> Hi,
> 
> did I see correctly and there is no OpenDocument format test suite yet?
> Like there is for SVG, e.g.
> www.w3.org/Graphics/SVG/WG/wiki/Test_Suite_Overview? One that checks the
> rendering or the internal document model or the functionality?

No there is no such thing.

> All I found are checkers for the file format produced, like Calligra's
> tools/scripts/validateODF.py.
> 
> What kind of automated/manual tests are there currently done for Calligra?
> E.g. before the current release?

There is a test set of ~1800 documents I run cstester on regularly to see if 
there are regressions.

> http://testsuite.opendocumentfellowship.com/ seems dead.
> http://opendocumentfellowship.com/validator tests just file correctness.
> http://officeshots.org/ seems to be for inter-program but not spec
> comparison (and misses Calligra?)

Maybe with the help of Cyrille I can this up for officeshots up with the 
released version. the ko server we use for the build bot could be used for it. 
michiel_l in the IRC channel is the guy behind this. I think it should be 
straight forward to set it up as there are already scripts for koffice we can 
that could be modified.

> What is the purpose of opendocumentfellowship.com and odfalliance.org, why
> do they not push for a test suite? (Calligra misses on
> http://opendocumentfellowship.com/applications)
> 
> I could e.g. imagine an automated roundtrip tester with pixel-by-pixel
> comparison of the rendering, tools/cstester/scripts/verifydocs.sh looks
> like that is done perhaps somewhere. But with which test documents?

There is already a roundtrip tester for calligra in cstester. 
cstester --roundtrip is what you are looking for.

I will send you the link for the test documents that we use to test calligra. 
They are downloaded randomly from the web.
> 
> The following seems to be a good start, but progress on these documents
> seems stuck?
> http://officeshots.org/galleries/view/opendocument-fellowship-test-suite
> 
> What would be the best place to discuss the creation of a useful test
> suite? Anyone going to that ODF Plugfest in Brussels next WE? Perhaps this
> could be a point to discuss there? To have a testing documents per each
> point of the ODF spec, with e.g. some rendering example if useful?

I'm going to the plugfest and as far as I know there are also others going 
there. There is also

www.oasis-open.org/committees/oic/

I can also discuss that in an ODF TC call after the plugfest but let me talk 
to some other people on the ODF TC before if it has been discussed before.

> For SVG rendering I could imagine a testing program which shows
> side-by-side the reference pixel image and what Calligra/Karbon loaded and
> is rendering. Is there something like that? Or how is SVG
> loading/rendering/saving tested?
> 
> Something similar might be doable for ODF.

cstester is your friend. It already has all you are looking for. However we 
miss svg files in the current test suite. If you provide me some svg files I 
would be happy to add them to the test documents.

> Same for other file formats: for XFig I could imagine to prepare a set of
> test files as well, covering all interesting aspects, for which I would
> take screenshots and do a program which shows the screenshot next to what
> is loaded and rendered.

works out of the box too with cstester as it can already load all files karbon 
can load.

> tools/slidecompare compares exactly what? loading directly vs. loading by
> conversion?

it can compare the content of stage to openoffice impress if I'm not mistaken.

There are also some tools to automate the stuff in the cstester subdirectories 
and a visualimagecompare to compare the results of 2 cstrunner/cstester runs 
visually to see what has changed.

Thorsten