[KPhotoAlbum] Import crash

[Robert Krawitz]

On Mon, 15 May 2017 22:46:55 +0200, Johannes Zarl-Zierl wrote:
> Hello Robert,
>
> Disclaimer: I don't have much time on my hand currently, so I can only review/
> approve/merge patches. Sorry for that...
>
> On Montag, 15. Mai 2017 09:31:41 CEST Robert Krawitz wrote:
>> On Mon, 15 May 2017 09:06:04 -0400 (EDT), Robert Krawitz wrote:
>> > On Mon, 15 May 2017 09:02:05 -0400 (EDT), Robert Krawitz wrote:
>> >> I'm trying to import a huge number of photos (as a way of syncing up
>> >> two image stores), and kpa crashed.  "Huge number" in this case means
>> >> 90,000; I tried a similar exercise with a much smaller number with no
>> >> trouble.
>
> That is quite an impressive number, if I may say so.
>
>> it happens after clicking Finish.
>> However, I was able to find the recursion.  It appears to actually be
>> recursion between copyNextFromExternal() and aCopyJobCompleted(); the
>> compiler notes the possibility for tail recursion from aCopyJobCompleted().
>
> So I guess many of the imported images are already in the database? This would lead to a direct recursion.
> As a quick fix I guess we could bypass the direct call to aCopyJobCompleted(0) in copyNextFromExternal() in the default case...

I did it by having aCopyJobCompleted() set (clear) a flag to indicate
whether to continue looping.  But there's a new problem: as the import
progresses, I get messages like this:

kf5.kio.core: Invalid URL: QUrl("img_6004_c2.jpg")
kf5.kio.core: Invalid URL: QUrl("img_6005-5.jpg")
kf5.kio.core: Invalid URL: QUrl("img_6006-5.jpg")
kf5.kio.core: Invalid URL: QUrl("img_6007-6.jpg")
kf5.kio.core: Invalid URL: QUrl("img_6008-5.jpg")
kf5.kio.core: Invalid URL: QUrl("img_6009-6.jpg")

That's because the export file creates unique filenames even if the
files live in different directories (I presume because if you actually
exported the images, rather than the file, they'd live in one
directory so would need unique filenames):

 <image dayFrom="4" minuteFrom="3" dayTo="4" yearTo="2017" secondFrom="54" height="3648" md5sum="d050e4efd71942bbee62c2b5562d488f" description="" monthFrom="2" file="img_5700-5.jpg" yearFrom="2017" width="5472" monthTo="2" angle="0" label="img_5700" hourFrom="14">
  <options>
   <option name="Folder">
    <value value="7dmk2/dcim/106eos7d"/>
   </option>
   <option name="Keywords">
    <value value="MIT-Coast Guard Men's Basketball 2017-02-04"/>
   </option>
   <option name="Media Type">
    <value value="Image"/>
   </option>
  </options>
 </image>

In the original index.xml, the schema is subtly different:

        <image file="7dmk2/dcim/106eos7d/img_5700.jpg" label="img_5700" startDate="2017-02-04T14:03:54" endDate="2017-02-04T14:03:54" angle="0" md5sum="d050e4efd71942bbee62c2b5562d488f" width="5472" height="3648">
            <options>
                <option name="Keywords">
                    <value value="MIT-Coast Guard Men's Basketball 2017-02-04"/>
                </option>
            </options>
        </image>

So this probably isn't going to do what I want after all.  Rats.

>> I might also note that during a (much smaller) import the timeline bar
>> constantly flashes, seemingly as each image is added to the database;
>> this resulted in my window manager crashing.
>
> Interesting. Report a bug with the window manager ;-)
> Honestly though, I guess we should do that more efficiently. It seems that ImportHandler::addNewRecord() calls DB::addImages for each image instead of assembling a list and adding the whole list at once...

I was looking at that, but I don't know whether it's safe to call
updateCategories() before addImages().  If not, this will have to be
done in two passes.

>> Finally, the process of comparing MD5 checksums against the database
>> appears to be extremely slow; it took hours to go through 90,000
>> images.  Perhaps the existing MD5 checksums need to be stored in a
>> hash, rather than a sequential comparison or something?
>
> Looking at DB::MD5Map, this is already a map. I'd have to look more closely to see what's eating cpu in there...
>
> Could you please open bug reports for these issues? Or better yet, provide patches ;-)

One of the problems is with displaying the images page.
createImagesPage() is fast enough, but when it Qt then goes to display
it, it's deathly slow if there are a lot of images (presumably there's
some inefficiency in Qt with creating a dialog with thousands of
items in it -- or the way KPA is doing it, since file browsers have to
be able to display lots of items).  Even if I have it not try to
display a pixmap and/or label, it's very slow.
-- 
Robert Krawitz                                     <rlk at alum.mit.edu>

***  MIT Engineers   A Proud Tradition   http://mitathletics.com  ***
Member of the League for Programming Freedom  --  http://ProgFree.org
Project lead for Gutenprint   --    http://gimp-print.sourceforge.net

"Linux doesn't dictate how I work, I dictate how Linux works."
--Eric Crampton