[KPhotoAlbum] Thumbnail Performance & Compatibility: ImageManager vs KIO

[Robert Krawitz]

On Sun, 30 Jun 2013 18:32:45 +0200, Johannes Zarl wrote:
> Hi Robert,
>
> Thanks for taking the time and giving me some feedback on this.
>
> On Saturday 29 June 2013 18:44:08 Robert Krawitz wrote:
>> ++ Performance, performance, and performance.  Oh, and performance while
>> we're at it.
>> The usual way of storing thumbnails as one image (png or jpg) per source
>> file is extremely inefficient.
>
> Ok, I take it that the 10th of a second that other programs need for
> loading the thumbnails is just too slow for you, then ;-)

Let's just say that it's noticeable.

> Seriously, though: I do understand that making KPA slower is just not an option. 
>
>> Reading each thumbnail requires a
>> context switch into the kernel at least, and if the file isn't in cache,
>> at least one I/O operation (it may require reading the directory in
>> addition to reading the file from disk).  Rotating media can't sustain
>> much more than 100 IOPS, which limits how quickly kpa can display
>> thumbnails.  In addition, there's some overhead from decoding the image
>> files.
>
> Maybe I'm wrong, but I thought that's also true of the current way
> things are done in KPA. After all, the thumbs-X file is opened and
> closed for every thumbnail that is being loaded. Context-switch-wise
> this should be about the same overhead as with multiple single
> thumbnails. And I'm also not so sure about file-system caching doing
> any good here either, since the current scheme performs extremely poor
> on network-mounts (Though probably not worse than a
> single-thumbnail-per-file scheme when the thumbnails directory is on
> the network share)

If that's what it does, it should be fixed.  I'm not sure that mmaping
the thumbnail files alone will help, since it will still be accessed
randomly, but it's one thing that could be tried.

>> (It also significantly reduces the storage required -- even though the
>> thumbnails are stored decoded, they take up fewer disk blocks, because
>> they avoid filesystem overhead for small files.)
>
> Actually, it roughly doubles the storage required. After all you have
> one copy of the slightly less efficiently stored thumbnails, used by
> almost all other applications, plus one copy of the slightly more
> efficiently stored thumbnails, used by KPA.

Since I typically don't use a lot of other applications on the large
majority of my image files, this isn't an issue (I currently have 1777
thumbnails in my .thumbnails, vs. something over 70,000 in KPA).  And
the storage is still small change compared to the image files
themselves.  At least for my purposes, a good tradeoff.  In terms of
actual storage, my KPA thumbnails consume about 450 MB; my entire image
directory is close to 700 GB.  The non-KPA thumbnails consume about 50
MB (in the .thumbnails/normal directory).

> If we already store the thumbnails twice, we could at least generate
> them only once. By using PreviewJob to create the thumbnail that we
> put into out thumbnailcache, we would effectively half the time spent
> on thumbnail generation.

Well, it would have to convert them into the canonical format, and do
the extra I/O to write them out.
-- 
Robert Krawitz                                     <rlk at alum.mit.edu>

MIT VI-3 1987 - Congrats MIT Engineers 5 straight men's hoops tourney
Tall Clubs International  --  http://www.tall.org/ or 1-888-IM-TALL-2
Member of the League for Programming Freedom  --  http://ProgFree.org
Project lead for Gutenprint   --    http://gimp-print.sourceforge.net

"Linux doesn't dictate how I work, I dictate how Linux works."
--Eric Crampton