[KPhotoAlbum] Thumbnail Performance & Compatibility: ImageManager vs KIO

[Robert Krawitz]

On Sun, 30 Jun 2013 19:19:16 +0200, Johannes Zarl wrote:
> On Sunday 30 June 2013 18:49:49 Robert Krawitz wrote:
>> > Actually, it roughly doubles the storage required. After all you have
>> > one copy of the slightly less efficiently stored thumbnails, used by
>> > almost all other applications, plus one copy of the slightly more
>> > efficiently stored thumbnails, used by KPA.
>> 
>> Since I typically don't use a lot of other applications on the large
>> majority of my image files, this isn't an issue (I currently have 1777
>> thumbnails in my .thumbnails, vs. something over 70,000 in KPA).  And
>> the storage is still small change compared to the image files
>> themselves.  At least for my purposes, a good tradeoff.  In terms of
>> actual storage, my KPA thumbnails consume about 450 MB; my entire image
>> directory is close to 700 GB.  The non-KPA thumbnails consume about 50
>> MB (in the .thumbnails/normal directory).
>
> Ok, that's valuable input for me. I don't want to change something
> just to realise afterwards that for most people(*) it's a non-issue or
> maybe even detrimental.
>
>> > If we already store the thumbnails twice, we could at least generate
>> > them only once. By using PreviewJob to create the thumbnail that we
>> > put into out thumbnailcache, we would effectively half the time spent
>> > on thumbnail generation.
>> 
>> Well, it would have to convert them into the canonical format, and do
>> the extra I/O to write them out.
>
> Well, yes. But if that's the only issue I'd say it's worth trying it
> out to see if it really is a problem.

That's fine, but fast thumbnail building is much more important to me
than shared thumbnails, which don't matter at all.

Just for reference, here's my general workflow:

1) Copy all files off memory cards into my image repository.

   I use an import script of my own to do this.  I have 4 cameras.  I
   shoot with continuous image numbering, and simply use the file
   names/directory hierarchy straight out of the camera (canonicalized
   to lower case, since recent versions of KDE/openSUSE don't allow me
   to specify mount options, but that's a story for another day).  This
   script actually parallelizes the copies, so if I have multiple cards,
   I can optimize the operation.

   Depending upon the shoot I shoot either JPEG or RAW+JPEG.  This has
   some other implications, which I'll mention below.

2) Read all image files into KPA.  This is a very time consuming step,
   so I want the read image files/build thumbnails to be as efficient as
   possible.

3) Apply a keyword to all images from the shoot.

4) Take a first pass through all images, flagging (with a token) all
   unusable images for immediate deletion.

   What I'd really like here is for tokens to apply to all images in a
   stack (RAW and JPEG, in my case).  Then I would first autostack all
   images, and the applied token would let me flag both of them in one
   shot.

   Perhaps typing tokens in upper case would apply to all members of the
   stack, vs. lower case only to the top image of the stack (which winds
   up being arbitrary in this case -- sometimes the top image is the RAW
   and sometimes it's the JPEG.  That's annoying, but I won't go there
   yet.

5) Delete all of the flagged images.

6) Start an rsync to sync the images to my other computer.  While
   that's taking place, I continue my other work.

7) Take a second pass through the images, identifying (with a different
   token) the ones I want to use (as opposed to others, which are
   acceptable but which I don't currently want to use).

8) Apply a "selection" keyword to the selected ones.  For example, if
   the event is named "Event 1", the selection keyword will be "Event 1
   selection".

9) Export the selection to a specified directory, using symlinks or
   copies as appropriate.

   This is another use case for being able to token all images in a
   stack.  Sometimes I want to export the RAW files, sometimes the
   JPEGs, and occasionally both.  If I could token (and then keyword)
   all images in the stack, I could simply select the selection, select
   the filename suffix I want, and do the export.

   Also note that I don't actually need the .kim file here.  It's not
   that big of a nuisance to delete it, but I don't use it.  Thing is,
   export is the operation closest to what I want, and I don't want to
   use dolphin or some other D&D file manager for that purpose.  I run
   kpa full screen, making it even less convenient,

Notice that up to this point, the only tools I've used are kpa itself
and a couple of scripts (which don't care about thumbnails).

>From here, there are several variants:

1) Batch upload to SmugMug.

   In this case, I export the images via copy.  I usually resize the
   images to half size, reduce the quality a bit, and use exiftran to
   canonicalize the rotation, and then upload.  I have a slow DSL link
   (1500/384); if I had "real" internet service, I'd probably skip this
   step.

2) Batch upload to Facebook.

   I resize the images to 800x800 or occasionally 1200x1200 at somewhat
   reduced quality, canonicalize the rotation (fb really doesn't get
   this right), and upload.  If I'm also uploading to SmugMug, I'll just
   upload to SmugMug and use SmugMug tools to create the Facebook album.

3) Build a DVD.

   Historically, I've used generate HTML in KPA to do this, and then
   edited the resulting HTML with a script to do some cleanup, and burn
   the DVD.  These days I don't do DVDs much.

4) Build a USB key.

   More recently, I simply dump off the images, add a copyright file and
   any other cover letter I want, and copy the result to the USB key.

5) Build one or more panoramas.

   In this case, I export by symlink, each pano sequence into a separate
   directory.  I then use Hugin to build the panorama, and edit the
   result in GIMP.  Sometimes I have to iterate the pano build before
   I'm ready to build the final result.

   After I'm done, I create an XCF file from the TIFF Hugin produced,
   and copy the entire directory back into my KPA repository so that the
   panorama will get indexed.  This is why I added an option to skip
   symlinks when reading files.

There are variations:

1) Sometimes I'll use RAW files, and sometimes JPEGs.

2) Sometimes I want to crop the files; I use gwenview for this.  While
   gwenview uses the shared thumbnail scheme, it's of no use here, since
   I've already copied the files out of my master tree.

3) Sometimes I use GIMP to edit some of the files; the thumbnails don't
   matter here.

-- 
Robert Krawitz                                     <rlk at alum.mit.edu>

MIT VI-3 1987 - Congrats MIT Engineers 5 straight men's hoops tourney
Tall Clubs International  --  http://www.tall.org/ or 1-888-IM-TALL-2
Member of the League for Programming Freedom  --  http://ProgFree.org
Project lead for Gutenprint   --    http://gimp-print.sourceforge.net

"Linux doesn't dictate how I work, I dictate how Linux works."
--Eric Crampton