Tiles data format

[Cyrille Berger]

On Tuesday 15 June 2010, Dmitry Kazakov wrote:
> >  So yes, in theory, it is more efficient to compress a set of tile than
> >  an
> > 
> > individual tile, since we would be able to use data from different tile
> > to find patterns.
> 
> In size - yes, but i'm not sure about the speed. Well, if you compress
> tiles one-by-one, with a window bigger than the size of a tile, you may
> get a good speed, but my guts say this speed will *not* be optimal.
> 
> I have an idea, that if we shrink the window (for example to half-size of a
> tile), we will not loose much in compression rate, but we can gain much in
> speed. That's an idea.
> 
> Grouping, tiles is not a good idea for swapping, you are right. But these
> results for tile-by-tile compression are quite strange, maybe, you used
> different compression algorithms for tile-by-tile vs entire compression,
> e.g. lzf vs zlib?
I used both:

* .kra with compressed tiles (lzf) and no global compression: 67MB, saving 
time 1.6s
* .kra with uncompressed tiles and global compression: 28MB, saving time 28s
* .kra with compressed tiles using deflate and no global compression: 27MB 
saving time 23s

> > 1) I don't see any reason of writing tile-save-compression system
> > 
> > > separately and before the one that will be used in a swapper. We'll
> > > just
> > > 
> > > make many code-duplications.
> > 
> > Apart from the compression method, I don't see many code-duplication (and
> > if you want my opinion, tile-save-compression is way more important than
> > swapper :) it is the last thing that truly annoys me in krita 2.2)
> 
> Well, first, there is an abstract compressor should be done first, to will
> allow swapper to save data in the same format as kra. And life of both of
> us will be much easier =)
I don't think there is an obligation of using the same algorithm for both 
solution. The requirement for saving is the fastest is the best.
 
> And second, i'm not sure we interpreted the results of those tests right.
> Sliding window size of zlib is configurable and it is big by default, so it
> could cause the difference.
>
> If we interpreted them wrong, then it's better to replace (configure)
> compression of the KoStore, than to do this change.
Except there is no API to do that (and it uses KDE stuff, so no easy change 
here). And assuming the ZIP file format even allows that.

> PS:
> Anyway, if you wish, i can hurry with this abstract compressor a bit. =)

class AbstrstractCompress { virtual compress(void* data, int data_length, 
void* output, int output_length) = 0; virtual decompress(void* data, int 
data_length, void* output, int output_length) = 0;   }; ? Or am I missing 
something ?

or compress(void* data, int data_length, void* output, int output_length, 
CompressionMethod)

Well anyway, the new format for tiles allows room for experimentation, so I 
will implement that, and then we can play with it.

-- 
Cyrille Berger