[KPhotoAlbum] (Re)building the EXIF database

[Robert Krawitz]

On Tue, 19 Jan 2016 00:42:23 +0100, Johannes Zarl-Zierl wrote:
> Hi Robert,
>
>> For each filename, it has to reconstruct the INSERT (which is a fixed
>> string), bind the values to it, and then exec the query.
>> [...]
>> I'd think that using execBatch() to batch up the inserts (maybe 100
>> images at a time) would be a lot faster.  Thoughts, anyone?
>
> Even without doing any profiling myself your analysis seems to be on the point. I'll try and see how much speedup a better solution will actually bring us.

92 IOPS works out to about 10.8 ms/IOPS.  The drive in question is a
5400 RPM 2 TB drive, so the average rotational latency is about 5.5
ms; 5.3 ms seek latency sounds about right.  So that, in combination
with seeing something in the range of 2-5% CPU utilization (on an
older processor -- i7-920XM), suggests very strongly that this is
indeed completely I/O-bound.

The numbers suggest that I/O and CPU would balance somewhere around
20-50 images per batch, although reading the metadata from the image
files will result in an irreducible number of I/O ops too.  If we
wanted to get really fancy, we might be able to use a scout thread to
preread the first 64K or whatnot of each image, but that would be for
a later stage.

I suspect batching this up in 100-200 images would work well.  I think
we can afford to bias in favor of performance over safety; if kpa
crashes, simply start the operation over.
-- 
Robert Krawitz                                     <rlk at alum.mit.edu>

***  MIT Engineers   A Proud Tradition   http://mitathletics.com  ***
Member of the League for Programming Freedom  --  http://ProgFree.org
Project lead for Gutenprint   --    http://gimp-print.sourceforge.net

"Linux doesn't dictate how I work, I dictate how Linux works."
--Eric Crampton