[KPhotoAlbum] Yet more thumbnail stuff

[Robert Krawitz]

It turns out that more threads make for better performance, in my case
up to about 2x the number of hyperthreads (so on my 4 core/8 thread
laptop, 16 threads seems to do best).  That's regardless of whether
I'm using my SSD or my hard disk.  On the SSD, both I/O and CPU more
or less saturate at about the same point, although that looks like
coincidence to me.  I was getting 80-90% CPU utilization and at times
over 500 MB/sec throughput:

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          88.00    0.00    2.28    9.54    0.00    0.18

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda               0.00         0.00         0.00          0          0
sdb               0.00         0.00         0.00          0          0
sdc               0.00         0.00         0.00          0          0
sdd             719.20    513540.80        28.00    2567704        140

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          88.19    0.00    2.21    9.58    0.00    0.03

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda               0.00         0.00         0.00          0          0
sdb               0.00         0.00         0.00          0          0
sdc               0.00         0.00         0.00          0          0
sdd             727.20    518520.00        24.80    2592600        124

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          85.61    0.00    2.11   11.95    0.00    0.33

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda               0.00         0.00         0.00          0          0
sdb               0.00         0.00         0.00          0          0
sdc               0.00         0.00         0.00          0          0
sdd             782.60    522548.80        30.40    2612744        152

On my trusty ol' rusty spinner, I got about 70 MB/sec, 100 IO/sec, and
16% CPU ute (max).  That doesn't quite look saturated in terms of I/O
(I should be able to get closer to 100 MB/sec, but I got better
numbers with 16 threads than with 8), while using less than a core:

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          16.52    0.00    1.58   71.74    0.00   10.17

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda               0.00         0.00         0.00          0          0
sdb             108.00     70941.60         0.00     354708          0
sdc               0.80         0.00         8.00          0         40
sdd               0.00         0.00         0.00          0          0

If I had an NVMe, I'd fully saturate the CPU.  If I had a much
faster/threadlier CPU, though, it might still stay in balance, but at
completely absurd levels.

--------

I notice that if thumbnails are being built while you exit from kpa,
it often segfaults.  This is more prevalent with the increase in
threads.  The problem is that the threads aren't being properly
cleaned up.  I came up with some fairly ugly logic whereby the main
window requests that the async loader exit.  The async loader clears
out the pending queue, sets a flag requesting exit, and sends each
thread a dummy image request with a special exiting flag.  I had to
have a few other places in the thumbnail building logic make the same
check and also arrange to send dummy requests.

But there was another crash I saw a couple of times, where during
thumbnail building kpa would crash for no particularly obvious reason
here:

(gdb) t 7
[Switching to thread 7 (Thread 0x7fffc4f9d700 (LWP 10535))]
#0  0x00000000005b2462 in DB::FileName::operator<(DB::FileName const&) const ()
(gdb) where
#0  0x00000000005b2462 in DB::FileName::operator<(DB::FileName const&) const ()
#1  0x0000000000567bd8 in ImageManager::ThumbnailCache::contains(DB::FileName const&) const ()
#2  0x000000000056ce4c in ImageManager::PreloadRequest::stillNeeded() const ()
#3  0x0000000000566c24 in ImageManager::RequestQueue::popNext() ()
#4  0x0000000000561aca in ImageManager::AsyncLoader::next() ()
#5  0x0000000000560eba in ImageManager::ImageLoaderThread::run() ()
#6  0x00007fffee285e0f in  () at /usr/lib64/libQt5Core.so.5
#7  0x00007fffecd31724 in start_thread () at /lib64/libpthread.so.0
#8  0x00007fffed676e8d in clone () at /lib64/libc.so.6

That looks like something killed off one or the other DB::FileName out
from under it, but I couldn't find just where that might have
happened.  The other stacks looked innocent enough to me.

But it does raise a question: why is the thumbnail cache a map rather
than a hash dictionary?  Do we need the ordering preserved or something?

-- 
Robert Krawitz                                     <rlk at alum.mit.edu>

***  MIT Engineers   A Proud Tradition   http://mitathletics.com  ***
Member of the League for Programming Freedom  --  http://ProgFree.org
Project lead for Gutenprint   --    http://gimp-print.sourceforge.net

"Linux doesn't dictate how I work, I dictate how Linux works."
--Eric Crampton