Tiles data format

[Cyrille Berger]

On Wednesday 16 June 2010, Dmitry Kazakov wrote:
> > * .kra with compressed tiles (lzf) and no global compression: 67MB,
> > saving time 1.6s
> 
> * .kra with uncompressed tiles and global compression: 28MB, saving time
> 28s
> 
> > * .kra with compressed tiles using deflate and no global compression:
> > 27MB saving time 23s
> 
> ^^ both 'deflate'?
> 
> I think it means that if we make a global compression of the whole data
> manager with lzf it will be the same 1.6s :)
And might be about the same compression rate. And is more complex and lose the 
possibility to read only certain tiles.

> Btw, are you going to disable KoStore compression only for data manager or
> for entire document? I think the latter one is not really fun.
Only for data manager.

> >  > And second, i'm not sure we interpreted the results of those tests
> > 
> > right.
> > 
> > > Sliding window size of zlib is configurable and it is big by default,
> > > so
> > 
> > it
> > 
> > > could cause the difference.
> > > 
> > > If we interpreted them wrong, then it's better to replace (configure)
> > > compression of the KoStore, than to do this change.
> > 
> > Except there is no API to do that (and it uses KDE stuff, so no easy
> > change here). And assuming the ZIP file format even allows that.
> 
> libz allows it for sure. See deflateInit2() in [1]. *
> 
> *[1] -  http://www.zlib.net/manual.html

Yes, but all ZIP coder/decoder I have seen call deflateInit2 with -MAX_WBITS as 
window bits.


> > > PS:
> > > Anyway, if you wish, i can hurry with this abstract compressor a bit.
> > > =)
> > 
> > class AbstrstractCompress { virtual compress(void* data, int data_length,
> > void* output, int output_length) = 0; virtual decompress(void* data, int
> > data_length, void* output, int output_length) = 0;   }; ? Or am I missing
> > something ?
> > 
> > or compress(void* data, int data_length, void* output, int output_length,
> > CompressionMethod)
> > 
> > Well anyway, the new format for tiles allows room for experimentation, so
> > I will implement that, and then we can play with it.
> 
> Something like that, but a bit more high-level. (do we need fast
> compression for the stuff other than tiles?) I want to encapsulate tile
> header into it. (the one that puts origin of the tile, it's size and etc).
> Something like:
> 
> class KisTileCompressor {
>     qint32 optimalBufferSize();     // for LZ77 - there is a formula for
> estimating high boundary for compressed size
>     void compressTile(KisTileSP tile, quint8* buffer);
>     KisTileSP decompressTile(quint8* buffer);
> };
> 
> In such a case i will use the same binary file format for swapper, this
> will allow simply copy swapper's data to the kra file when saving.
> 
> Hm... wait! Your idea is even better in some way! We need only one
> KisTileCompressor that uses different abstract compress engines. Yeah. You
> are right with idea of KisAbstractCompressor. But it should provide a
> couple of additional interfaces:
> 
> virtual qint32 estimateCompressedSize(qint32 sourceSize); // <- to be able
> to compress tiles in one pass.
> virtual void adjustForDataSize(qint32 tileDataSize); // <- this one will
> configure sliding window size (and friends) for particular data-stream-size
> 
> What do you think?

I don't see the point of optimalBufferSize ? or estimateCompressedSize ? In our 
case the buffer should be the size of the tile, and the data is saved 
compressed only if the compression result is inferior to the size of the tile. 
I don't see the point in making it more complicated.

Also I don't know how the swapper will work, but for the loading of a file it 
is more convenient to have decompressTile(KisTileSP, quint8* buffer); (but then 
I would pass data pointer to have an interface that can be easilly used 
outside of the context :) )

-- 
Cyrille Berger