Regressions || Huston we have a problem

[Sebastian Sauer]

On Saturday 16 July 2011 15:44:52 Pierre wrote:
> On Friday 15 July 2011 12:30:01 Sebastian Sauer wrote:
> > Aloha,
> > 
> > yesterday at the ODF plugfest 2011 (
> > http://www.opendocsociety.org/news/2011- berlin-plugfest/ ) we had two
> > interoperability tests and failed on both of them. It seems both where
> > working at some point but cause of regressions didn't any longer at the
> > event. That is a problem.
> > 
> > Let me add that the problem are NOT the regressions. That can happen. The
> > problem is that we did not discover them for more then a month till that
> > event.
> > 
> > So, the question is how to improve that situation to make sure we are
> > able to discover regressions faster?
> > 
> > I see two ways;
> > 1. unittests also for saving.
> > 2. cstester roundtrips.
> > 
> > The first would be optimal but it would need *lot* of time especially if
> > we try to get a good coverage done.
> > 
> > The second is the fastest way (I can think of atm). We could first fix
> > cstester so document-roundtrips are proper working and second move that
> > to a server that runs cstester against the large collection of documents
> > located on the KDE-svn server in the kofficetests directory. Maybe we
> > can run that once a day in an automated way and then incoperate that
> > into our IRC unittest-bot? Or maybe we can provide a webpage that shows
> > in which ways what documents changed from one day to the other?
> > 
> > What do you think? Is there maybe a better way? Maybe even an easier way?
> > Or....?
> 
> Hi
> 
> Really bad news indeed.
> What kind of regressions are we looking at ?
> Is it "global" regressions, affecting for instance the whole document
> content, or just "macro" regressions like a property not being saved
> properly anymore ? IMHO, both should be covered by different unit tests
> for better efficiency...

In an ideal world all kind of regressions. But in a realistic world I would 
say only major regressions where we e.g. crash on loading or saving or lose 
(important) content on loading/saving or end in loops or...

And yes, we definitivly should have unittest for this too. My idea would be to 
add a Roundtrip-class to calligra/tools/roundtriptests/ that provides the 
functionality to load+save+reload content in Words/Tables/Stage and then write 
single unittests to test for specific concrete cases. I started with that on 
the weekend but it will take some more time till the first unittest works as 
expected.

Or maybe there is a better/easier way to unittest roundtrips?