[KPhotoAlbum] Import crash

[Johannes Zarl-Zierl]

On Dienstag, 16. Mai 2017 21:54:01 CEST Robert Krawitz wrote:
> On Mon, 15 May 2017 22:46:55 +0200, Johannes Zarl-Zierl wrote:
> > 
> > So I guess many of the imported images are already in the database? This
> > would lead to a direct recursion. As a quick fix I guess we could bypass
> > the direct call to aCopyJobCompleted(0) in copyNextFromExternal() in the
> > default case...
> I did it by having aCopyJobCompleted() set (clear) a flag to indicate
> whether to continue looping.  But there's a new problem: as the import
> progresses, I get messages like this:
> 
> kf5.kio.core: Invalid URL: QUrl("img_6004_c2.jpg")
> kf5.kio.core: Invalid URL: QUrl("img_6005-5.jpg")
> kf5.kio.core: Invalid URL: QUrl("img_6006-5.jpg")
> kf5.kio.core: Invalid URL: QUrl("img_6007-6.jpg")
> kf5.kio.core: Invalid URL: QUrl("img_6008-5.jpg")
> kf5.kio.core: Invalid URL: QUrl("img_6009-6.jpg")
> 
> That's because the export file creates unique filenames even if the
> files live in different directories (I presume because if you actually
> exported the images, rather than the file, they'd live in one
> directory so would need unique filenames):

[snip]

> So this probably isn't going to do what I want after all.  Rats.

Just to be clear about the problem: You end up with a bunch of non-existing 
images in the database, in addition to the original that is already in the 
database?

If so, maybe you can help things along by using the "Find duplicates" dialog.

A proper fix needs to change the way we create export files. In the past, I 
was always hesitant to touch the .kim file format in order to avoid 
compatibility problems. Maybe it's time to revisit this decision...


> >> I might also note that during a (much smaller) import the timeline bar
> >> constantly flashes, seemingly as each image is added to the database;
> >> this resulted in my window manager crashing.
> > 
> > Interesting. Report a bug with the window manager ;-)
> > Honestly though, I guess we should do that more efficiently. It seems that
> > ImportHandler::addNewRecord() calls DB::addImages for each image instead
> > of assembling a list and adding the whole list at once...
> I was looking at that, but I don't know whether it's safe to call
> updateCategories() before addImages().  If not, this will have to be
> done in two passes.

Just looking at the code I would think that it is safe to change 
updateCategories() so that it accepts an ImageInfoList and call both 
addImages() and updateCategories() once after assembling the image list.


> >> Finally, the process of comparing MD5 checksums against the database
> >> appears to be extremely slow; it took hours to go through 90,000
> >> images.  Perhaps the existing MD5 checksums need to be stored in a
> >> hash, rather than a sequential comparison or something?
> > 
> > Looking at DB::MD5Map, this is already a map. I'd have to look more
> > closely to see what's eating cpu in there...

> One of the problems is with displaying the images page.
> createImagesPage() is fast enough, but when it Qt then goes to display
> it, it's deathly slow if there are a lot of images (presumably there's
> some inefficiency in Qt with creating a dialog with thousands of
> items in it -- or the way KPA is doing it, since file browsers have to
> be able to display lots of items).  Even if I have it not try to
> display a pixmap and/or label, it's very slow.

I would assume that file managers don't need run into this issue because most 
file systems perform poorly when you stash thousands of files into a single 
directory.
For widgets that really have a huge number of items, I guess you could do 
progessive loading? I haven't looked into this, though...

Cheers,
  Johannes