[KimDaBa] Multithreaded image loader?

[Robert L Krawitz]

   From: "Jesper K. Pedersen" <blackie at blackie.dk>
   Date: Mon, 10 Jan 2005 20:50:46 +0100

   On Monday 10 January 2005 14:24, Robert L Krawitz wrote:
   | I notice a comment in the image manager that the image loader can't be
   | made multithreaded because the JPEG loader isn't MT-safe.  Is this an
   | issue in the JPEG library or somewhere else?  Particularly (but not
   | exclusively) on an SMP/CMP or HT system, multithreading the loader
   | might offer a significant performance improvement.

   I removed the mutex from the code, and tries starting multiple
   image loaders, but I couldn't really tell any difference.

   To try it out, cvs up to HEAD, and apply this patch
   Let me know if you see any significant difference.

So I don't believe this to be my performance problem, after all.

I was trying to figure out how to test it, and I decided that one way
would be to time how long it took to scroll all the way from the top
to the bottom of all 6800 of my photos, by simply holding down the
page down key.  It ran slower and slower (the scrollbar moved slower
and slower), and eventually ground to a halt.  At that point, I found
my machine had ground to a halt!  I eventually managed to get a root
shell, and found that my kimdaba process had grown to 800 MB,
consuming all of my swap along the way.  Unfortunately, I wasn't able
to put a debugger on it.

What it *looks* like is happening is that the load queue is getting
badly backed up, and somehow everything then goes wrong.  I do notice
that changing the priority to ImageManager::load() from low to high
does make the scrolling considerably more responsive.  It looks
strange, since the images are filled in from bottom to top and right
to left, but it does result in the thumbnails being filled in much
more quickly.

The other thing I noticed, by running top, is that when I hold down
the page down button the memory consumption rapidly goes up.  As soon
as I release it, it drops off and eventually ends up back at the
baseline.

Also, it's very important to do the loading asynchronously, as you did
(the second call to pixmapCache().find() forces this to all happen
synchronously, which kills performance in this situation).

So what's probably needed here is a way to cancel a load in progress;
if that can't be done, perhaps there should be a queue limit beyond
which things are dequeued, and can be re-queued if need be.

diff -u thumbnail.cpp{~,}
--- thumbnail.cpp~      2005-01-08 19:31:51.000000000 -0500
+++ thumbnail.cpp       2005-01-10 20:50:49.987471859 -0500
@@ -227,7 +227,7 @@
         return pix;
 
     int size = Options::instance()->thumbSize();
-    ImageManager::instance()->load( _imageInfo->fileName(),  const_cast<ThumbNail*>( this ), _imageInfo->angle(), size, size, true, false );
+    ImageManager::instance()->load( _imageInfo->fileName(),  const_cast<ThumbNail*>( this ), _imageInfo->angle(), size, size, true, true );
     if ((pix = pixmapCache().find( _imageInfo->fileName())) != NULL)
         return pix;
     else


-- 
Robert Krawitz                                     <rlk at alum.mit.edu>

Tall Clubs International  --  http://www.tall.org/ or 1-888-IM-TALL-2
Member of the League for Programming Freedom -- mail lpf at uunet.uu.net
Project lead for Gimp Print   --    http://gimp-print.sourceforge.net

"Linux doesn't dictate how I work, I dictate how Linux works."
--Eric Crampton