Is there an accepted/official ODF test suite?

[Friedrich W. H. Kossebau]

Hi Thorsten,

Am Freitag, 13. April 2012, 06:19:04 schrieb Thorsten Zachmann:
> Hello,
> 
> On Thursday, April 12, 2012 20:05:51 Friedrich W. H. Kossebau wrote:
> > Hi,
> > 
> > did I see correctly and there is no OpenDocument format test suite yet?
> > Like there is for SVG, e.g.
> > www.w3.org/Graphics/SVG/WG/wiki/Test_Suite_Overview? One that checks the
> > rendering or the internal document model or the functionality?
> 
> No there is no such thing.
> 
> > All I found are checkers for the file format produced, like Calligra's
> > tools/scripts/validateODF.py.
> > 
> > What kind of automated/manual tests are there currently done for Calligra?
> > E.g. before the current release?
> 
> There is a test set of ~1800 documents I run cstester on regularly to see if
> there are regressions.

Sounds quite large :) But they are not run automatically I guess, so anybody 
with some free time could take a look at the results to see if there is 
something she/he could work on, right?

At least I cannot find anything related on http://quality.calligra.org/ but 
than this might be due to the fact that these documents are not public domain 
or licensed in a similar suitable way.

Perhaps the "OpenDocument Fellowship Test Suite" as used e.g. for
http://officeshots.org/galleries/view/opendocument-fellowship-test-suite
could be used here, as "released under the Creative Commons Attribution 2.5 
License.". There is also a Revised version, but that does not talk about its 
license.

I just can say that it would be very cool to have the document based tests run 
automatically and the results (incl. rendered images) available in public on 
some server, so spare-time contributors do not have to use their time and do 
the work to run the whole tests themselves, but can just browse the results 
and pick those documents where they see an issue to scratch and spend the time 
on scratching instead </wish> :)

Would also be a good show-case for the power of the Calligra engine.

> > http://testsuite.opendocumentfellowship.com/ seems dead.
> > http://opendocumentfellowship.com/validator tests just file correctness.
> > http://officeshots.org/ seems to be for inter-program but not spec
> > comparison (and misses Calligra?)
> 
> Maybe with the help of Cyrille I can this up for officeshots up with the
> released version. the ko server we use for the build bot could be used for
> it. michiel_l in the IRC channel is the guy behind this. I think it should
> be straight forward to set it up as there are already scripts for koffice
> we can that could be modified.

Nothing I will put myself behind to make this happen. But I think it would be 
good for Calligra if that is done :) (both for making Calligra present and 
getting feedback where Calligra stands in terms of proper ODF support)

> > What is the purpose of opendocumentfellowship.com and odfalliance.org, why
> > do they not push for a test suite? (Calligra misses on
> > http://opendocumentfellowship.com/applications)
> > 
> > I could e.g. imagine an automated roundtrip tester with pixel-by-pixel
> > comparison of the rendering, tools/cstester/scripts/verifydocs.sh looks
> > like that is done perhaps somewhere. But with which test documents?
> 
> There is already a roundtrip tester for calligra in cstester.
> cstester --roundtrip is what you are looking for.

Cool, so I expected right from the Calligra devs :)

> I will send you the link for the test documents that we use to test
> calligra. They are downloaded randomly from the web.

Thanks for having me that link. Perhaps I might find time to make use of these 
documents.

> > The following seems to be a good start, but progress on these documents
> > seems stuck?
> > http://officeshots.org/galleries/view/opendocument-fellowship-test-suite
> > 
> > What would be the best place to discuss the creation of a useful test
> > suite? Anyone going to that ODF Plugfest in Brussels next WE? Perhaps this
> > could be a point to discuss there? To have a testing documents per each
> > point of the ODF spec, with e.g. some rendering example if useful?
> 
> I'm going to the plugfest and as far as I know there are also others going
> there. There is also
> 
> www.oasis-open.org/committees/oic/
> 
> I can also discuss that in an ODF TC call after the plugfest but let me talk
> to some other people on the ODF TC before if it has been discussed before.

Good. I will finally tonight compile the issues I have found so far with ODF 
and will send them to you (and else?) hoping you will have the opportunity to 
push them into the discussion.

> > For SVG rendering I could imagine a testing program which shows
> > side-by-side the reference pixel image and what Calligra/Karbon loaded and
> > is rendering. Is there something like that? Or how is SVG
> > loading/rendering/saving tested?
> > 
> > Something similar might be doable for ODF.
> 
> cstester is your friend. It already has all you are looking for. However we
> miss svg files in the current test suite. If you provide me some svg files I
> would be happy to add them to the test documents.

I guess simply using the files from the SVG Test Suite would be a good start, 
at least the ones not testing animation. Those provide both simple SVG files 
as well as reference PNG files.

The whole test suite data can be downloaded e.g. by this link:
http://www.w3.org/Graphics/SVG/Test/20110816/archives/W3C_SVG_11_TestSuite.tar.gz

I guess it needs someone to step through these files and see which ones are 
suitable for the SVG import filter (I guess SVG scripts will not be supported 
soon, right?). So there could be a script which downloads the archive and 
installs those files from the created list which are useful.

Something similar could be done with the MathML test suite that Jos mentioned. 
Hm, could not find an archive of the test suite (anymore), perhaps use wget to 
pull the test data :)

> > Same for other file formats: for XFig I could imagine to prepare a set of
> > test files as well, covering all interesting aspects, for which I would
> > take screenshots and do a program which shows the screenshot next to what
> > is loaded and rendered.
> 
> works out of the box too with cstester as it can already load all files
> karbon can load.

So once I played enough with cstester to get the idea how it works, would you 
accept the test XFig files I am going to create and have them put in
http://websvn.kde.org/trunk/tests/calligratests/import/xfig
?

> > tools/slidecompare compares exactly what? loading directly vs. loading by
> > conversion?
> 
> it can compare the content of stage to openoffice impress if I'm not
> mistaken.
> 
> There are also some tools to automate the stuff in the cstester
> subdirectories and a visualimagecompare to compare the results of 2
> cstrunner/cstester runs visually to see what has changed.

Thanks for the pointers, now playing around with those
Friedrich