[KPhotoAlbum] More performance work
rlk at alum.mit.edu
Fri May 11 03:47:13 BST 2018
I've finally managed to completely eliminate the slowdown associated
with autostacking images at load time, as well as eliminating and/or
hiding a lot of I/O latency. This involves four things:
1) Full transaction semantics around EXIF database insertions: start a
transaction at the start of a batch load, commit it or roll it back
as appropriate when the load finishes/is aborted. I originally did
this to eliminate some I/O in the case of very big loads (big
enough to blow out your physical RAM, so the insertion would need a
second pass over the files to grab the EXIF data). However, it
also simplifies things somewhat and may allow us to build
thumbnails concurrently with loading images, which would reduce I/O
(for the same reason) and possibly allow us to get better
2) Use a scout thread to read the images. The scout thread simply
loops over all of the images being loaded and opens the files,
reads them in full, and closes them. If they're already in RAM the
scout thread runs very quickly; if they're not, it runs ahead of
the reader and pulls the files in so that the loader doesn't have
to wait for physical I/O to complete. On a very fast machine with
very slow I/O (NFS in particular) it probably won't be able to keep
ahead of the loader, but there's not a lot we can do. I found that
it significantly improves performance and CPU utilization because
with conventional drives on a reasonably fast processor (quad core
2.8->3.7 GHz) it now gets 100% CPU utilization.
On a slow system, if the scout thread gets too far ahead, it could
actually result in memory contention and extra I/O. If that were
to prove a real world problem we could look into what to do about
it, but it would make the code rather more complex (with
synchronization needed between the scount and the reader).
3) Reverse cache the MD5 checksums. Currently kpa only keeps an
MD5->filename mapping, so when it does the stack check based on
filename it has to recompute the MD5 checksum of the stackee to
ensure that it's not a duplicate. That means lots of I/O,
particularly in my case where I'm usually stacking a new smaller
edited file on top of the full size file. Caching the reverse
mapping eliminates that I/O (and the not entirely free MD5
calculation, although on spinning rust the I/O is the bad part).
4) Strengthening the image info cache so that there's a filename to
ImageInfo hash static to XMLDB/Database.cpp rather than just one
function within it. That's not quite doing what I expect it to do,
so I'm working on it.
I could probably separate out some of these pieces, but they all touch
LoadExtraFile. Right now the code is a mess, with a lot of debugging
code still left, the indentation all over the place, and stuff like
that, but I could generate a patch if people want to review it even in
a rather crude state.
I know I've been a bit single minded about the load performance, but
that's really the very first thing people see when they look at kpa or
any similar program. I know that when I've played with Digikam the
image load is so slow that it makes even experimenting painful.
Loading 3000 images in something under 3 minutes is fast enough that
even quite large image collections can be tested out without too much
discomfort. I'd honestly like to be able to load such small images (1
MB) even faster than the 16-18 images/second I'm getting, but that
will require a closer look at what's going on.
Robert Krawitz <rlk at alum.mit.edu>
*** MIT Engineers A Proud Tradition http://mitathletics.com ***
Member of the League for Programming Freedom -- http://ProgFree.org
Project lead for Gutenprint -- http://gimp-print.sourceforge.net
"Linux doesn't dictate how I work, I dictate how Linux works."
More information about the Kphotoalbum