Tiles data format

[Dmitry Kazakov]

He-he :) Right at the moment i was planning swapping and compression
architecture =).

Well, i have several points:
1) If we change the format, then the first thing we should do is add VERSION
field to the beginning of the file. Just to make our lives easier.

2) I was considering swapping, and i wanted to use LZO compression algorithm
for this. According to some papers it is the fastest. (But those papers
didn't cover LZF). [0]

3) For swapping it will surely be *not* efficient to work with a single
tile. So i was going to join them into slabs and transport to disk at once,
though compression will most likely be by-tile.

4) The thing that worries me the most is "compression sliding window". This
term relates to most of LZx algorithms. LZO uses ~8-64KiB (do not know
exactly). Regular tile has a size of 4*64*64=16KiB. It means that the tile
can be fit into this window fully. This (not sure) means that the algorithm
will work less efficient(!), than it can.

We must understand this effect fully before writing anything, or we can get
many problems later.


More than that, effect you were exploring in March can (not sure, again) be
a consequence of this. Original LZ77 algorithm does not depend on the size
of input and almost linear in time (when the size of the input is much
bigger than the size of the sliding window).


If that is so, this non-linearity can be a flaw of libLZF. Maybe, some
post-processing with Haffman trees.

[0] - http://www.oberhumer.com/opensource/lzo/


My conclusion:
1) I don't see any reason of writing tile-save-compression system separately
and before the one that will be used in a swapper. We'll just make many
code-duplications.

2) We can make kra-format more extensible before this. Yes. And if we define
new format well I can easily fix KisTiledDataManager::write/read
accordingly.



On Sun, Jun 13, 2010 at 4:36 PM, Cyrille Berger <> wrote:

>
> TILESNUMBER 10
> COMPRESSIONSCHEME NONE
> DATA
> 0,0,64,64
>

^^^ What is in this line? Is there any difference with the data after ']'?
And why there are additional digits appear when the compression is tured on?



> [64x64xNxd binarydata]0,64,64,64
>

^^^ What is 'Nxd'? pixels*depth?


> [64x64xNxd binarydata]0,64,128,64
> ...
> [64x64xNxd binarydata]
>


Actually, this is a good idea. I think for future use we need to allow
different tilesize as well. So i would modify it like that:

VERSION 2
TILEWIDTH 64
TILEHEIGHT 64
PIXELSIZE 4
TILESNUMBER 10
COMPRESSIONSCHEME LZF
DATA
0,0,64,64,12314
[12314 bytes of binarydata]
64,64,64,64,3333
[3333 bytes of binarydata]

I don't see a reason of saving width and height of every tile. Maybe, omit
it? And make like:

VERSION 2
TILEWIDTH 64
TILEHEIGHT 64
PIXELSIZE 4
TILESNUMBER 10
COMPRESSIONSCHEME LZF
DATA
0,0,12314
[12314 bytes of binarydata]
64,64,3333
[3333 bytes of binarydata]


Pixel size will be used for asserts mostly.



-- 
Dmitry Kazakov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.kde.org/pipermail/kimageshop/attachments/20100613/ebfeb648/attachment.htm