Failing to load collection

[Sigfrid Lundberg]

OK, I'll try to report to the relevant bug report systems. I'm not
sure my observations helps very much, though :^/

I've upgraded my exiv2 to the tar.gz I found on their site, 0.26 something.
Removed my previous one with apt-get remove.

Now 'exiv2 -V'  gives 'exiv2 0.26 001a00 (64 bit build)'

The following produces exactly one warning per JPG, but not a single exception:

find 2011-2012-freddy/ -name '*.JPG' -exec exiv2 -pa {} > /dev/null \;

If I instead execs exiv2 for each file, regardoess, it throws
exceptions on AVI files, like

Exiv2 exception in print action for file 2011-2012-freddy/P1170859.AVI:
2011-2012-freddy/P1170859.AVI: The file contains data of an unknown image type

Likewise, if I run exiv2 on each file in the directory where it
stopped when I first contacted this list

find 201112-201112/ -type f -exec exiv2 -pa {} > /dev/null \;

Exiv2 exception in print action for file 201112-201112/R0010678.DNG.pp3:
201112-201112/R0010678.DNG.pp3: The file contains data of an unknown image type
Error: XMP Toolkit error 203: Duplicate property or field node
Warning: Failed to decode XMP metadata.

Where pp3 is a file left by rawtherapee. Then we haven't even come to
all the directories where I mix text image and sounds and whatever.
eps, pdf, ms, latex, html....

Is the combo exiv2 and digikam now expecting absolutely clean
directories untouched by text and video editors and no soundfiles in
the corners?

Sigge








On 6 June 2017 at 16:18, Gilles Caulier <caulier.gilles at gmail.com> wrote:
> Et voilà... an Exiv2 exception is generated with a specific file.
>
> Now, try to identify which file create this dysfunction. It sound like
> a file included in /home/sigge/Dropbox/galleri/2011-2012-freddy/
>
> With the Exiv2 CLI tool, try to just read each file from this album to
> see if something is wrong while scanning tag contents. Warning :
> digiKam 5.6.0 Appimage use the last stable Exiv2 0.26, and previous
> Exiv2 0.25 can not generate an exception).
>
> When you found the right file, report the problem to Exiv2 bugzilla
> for future investigations. Don't forget to share the problematic file.
>
> From the digiKam side, it still a problem to wrap the exception from
> Exiv2 without to lock the application. Typically a time out when the
> item is scanned to a separated thread. The Exiv2 exception is catch
> properly, but something is wrong somewhere to complete the scan even
> if an error occur.
>
> For this point, please open a file in KDE bugzilla, in
> digiKam/metadata section with a link to the suspected file to hack
> digiKam core implementation.
>
> Gilles Caulier
>
> 2017-06-06 16:00 GMT+02:00 Sigfrid Lundberg <sigfrid at sigfrid-lundberg.se>:
>> That directory is JPG only, taken with an Olympus, no DNG
>> Exceptions I do understand. So close to java ;^)
>>
>> GDB supports the exiv2 hypothesis, though. GDB claims the following
>>
>> Thread 5 "Digikam::ScanCo" hit Catchpoint 1 (exception thrown),
>> 0x00007fffee1848bd in __cxa_throw ()
>>    from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
>> (gdb) bt
>> #0  0x00007fffee1848bd in __cxa_throw () from
>> /usr/lib/x86_64-linux-gnu/libstdc++.so.6
>> #1  0x0000003455b53b15 in Exiv2::ImageFactory::open (path=...,
>> useCurl=<optimized out>)
>>     at
>> /b/ext_exiv2/ext_exiv2-prefix/src/ext_exiv2/exiv2-trunk/src/image.cpp:848
>> #2  0x00007ffff6775c71 in Digikam::MetaEngine::load
>> (this=this at entry=0x7fffd4063c60, filePath=...)
>>     at /b/dktemp/digikam-master/core/libs/dmetadata/metaengine.cpp:281
>> #3  0x00007ffff67be5c6 in Digikam::DMetadata::load
>> (this=this at entry=0x7fffd4063c60, filePath=...)
>>     at /b/dktemp/digikam-master/core/libs/dmetadata/dmetadata.cpp:96
>> #4  0x00007fffee59a3fa in Digikam::ImageScanner::loadFromDisk
>> (this=this at entry=0x7fffe13077e0)
>>     at
>> /b/dktemp/digikam-master/core/libs/database/item/imagescanner.cpp:1563
>> #5  0x00007fffee59a590 in Digikam::ImageScanner::newFile
>> (this=this at entry=0x7fffe13077e0, albumId=2)
>>     at /b/dktemp/digikam-master/core/libs/database/item/imagescanner.cpp:289
>> #6  0x00007fffee4dbb7e in Digikam::CollectionScanner::scanNewFile
>> (this=this at entry=0x7fffe1307bf0,
>>     info=..., albumId=2)
>>     at
>> /b/dktemp/digikam-master/core/libs/database/collection/collectionscanner.cpp:1283
>> #7  0x00007fffee4dd488 in Digikam::CollectionScanner::scanAlbum
>> (this=this at entry=0x7fffe1307bf0,
>>     location=..., album=...)
>>     at
>> /b/dktemp/digikam-master/core/libs/database/collection/collectionscanner.cpp:1106
>> #8  0x00007fffee4ddaf9 in Digikam::CollectionScanner::scanAlbumRoot (
>>     this=this at entry=0x7fffe1307bf0, location=...)
>>     at
>> /b/dktemp/digikam-master/core/libs/database/collection/collectionscanner.cpp:835
>> #9  0x00007fffee4ddd26 in Digikam::CollectionScanner::completeScan
>> (this=this at entry=0x7fffe1307bf0)
>>     at
>> /b/dktemp/digikam-master/core/libs/database/collection/collectionscanner.cpp:488
>> #10 0x00007ffff74f485e in Digikam::ScanController::run (this=
>>     0x7ffff7dd4da0
>> <_ZZN7Digikam12_GLOBAL__N_113Q_QGS_creator13innerFunctionEvE6holder>)
>>     at
>> /b/dktemp/digikam-master/core/libs/database/utils/scancontroller.cpp:708
>> #11 0x00000034890aef49 in ?? () from
>> /tmp/.mount_sIj2qg/usr/lib/libQt5Core.so.5
>> #12 0x00007ffff5f1d6ba in start_thread (arg=0x7fffe1308700) at
>> pthread_create.c:333
>> ---Type <return> to continue, or q <return> to quit---
>> #13 0x00007fffed91582d in clone () at
>> ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
>> (gdb)
>>
>>
>> On 6 June 2017 at 15:51, Gilles Caulier <caulier.gilles at gmail.com> wrote:
>>>
>>> Well it simple. digiKam is stopped in debugger due to a C++ exception.
>>> You get the GDB prompt and it wait a command to hack.
>>>
>>> Just enter "bt" for backtrace to catch the full backtrace, and
>>> especially where the program is stopped in source code.
>>>
>>> The notice is explained in details here :
>>>
>>> https://www.digikam.org/contribute/
>>>
>>> Note : I'm 90% sure that you have a problem with Exiv2 with a DNG
>>> file... (:=)))...
>>>
>>> Gilles Caulier
>>>
>>> 2017-06-06 15:43 GMT+02:00 Sigfrid Lundberg <sigfrid at sigfrid-lundberg.se>:
>>> > Thanks for rapid response...  Starting with
>>> > digikam-5.6.0-01-x86-64.appimage
>>> > I did with debug option
>>> >
>>> > 1. I created an empty directory (~sigge/digikam) for the db
>>> > 2 Tried with the very first directory inside my gallery area, which
>>> > contains
>>> > a some 120 JPGs.
>>> >
>>> > I then got to 85% of that directory, with the following output from GDB
>>> > and
>>> > digikam:
>>> >
>>> > digikam.dimg:
>>> > "/home/sigge/Dropbox/galleri/2011-2012-freddy/P1080855.JPG"  :
>>> > JPEG file identified
>>> > digikam.database: Adding new item
>>> > "/home/sigge/Dropbox/galleri/2011-2012-freddy/P1080855.JPG"
>>> > digikam.metaengine: DateTime => Exif.Photo.DateTimeOriginal =>
>>> > QDateTime(2012-01-08 16:43:57.000 CET Qt::TimeSpec(LocalTime))
>>> > digikam.metaengine: DateTime (Exif digitalized):  s�n jan. 8 16:43:57
>>> > 2012
>>> > digikam.metaengine: Orientation => Exif.Image.Orientation =>  8
>>> > digikam.database: Scanning took 10 ms
>>> > digikam.database: Finishing took 0 ms
>>> > digikam.metaengine: Exiv2 ( 2 ) :  Directory OlympusCs, entry 0x0101:
>>> > Strip
>>> > 0 is outside of the data area; ignored.
>>> >
>>> > digikam.dimg:
>>> > "/home/sigge/Dropbox/galleri/2011-2012-freddy/P1080856.JPG"  :
>>> > JPEG file identified
>>> > digikam.database: Adding new item
>>> > "/home/sigge/Dropbox/galleri/2011-2012-freddy/P1080856.JPG"
>>> > digikam.metaengine: DateTime => Exif.Photo.DateTimeOriginal =>
>>> > QDateTime(2012-01-08 16:44:02.000 CET Qt::TimeSpec(LocalTime))
>>> > digikam.metaengine: DateTime (Exif digitalized):  s�n jan. 8 16:44:02
>>> > 2012
>>> > digikam.metaengine: Orientation => Exif.Image.Orientation =>  8
>>> > digikam.database: Scanning took 12 ms
>>> > digikam.database: Finishing took 0 ms
>>> > [Switching to Thread 0x7fffe1308700 (LWP 16323)]
>>> >
>>> > Thread 5 "Digikam::ScanCo" hit Catchpoint 1 (exception thrown),
>>> > 0x00007fffee1848bd in __cxa_throw ()
>>> >    from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
>>> > (gdb)
>>> >
>>> >
>>> > I still have it running, so if you have suggestions for next step, I can
>>> > continue!
>>> >
>>> > Sigge
>>> >
>>> >
>>> >
>>> >
>>> > On 6 June 2017 at 14:54, Gilles Caulier <caulier.gilles at gmail.com>
>>> > wrote:
>>> >>
>>> >> Hi,
>>> >>
>>> >> I'm not sure if the DNG file identified is really the problem.
>>> >>
>>> >> But it's clear that something is wrong with a file scanned to populate
>>> >> the database.
>>> >>
>>> >> I suspect a problem with Exiv2, which is mostly used to scan image
>>> >> properties in background.
>>> >>
>>> >> So the best way to identify the problem, especially the image file, is
>>> >> :
>>> >>
>>> >> 1/ use only the last 5.6.0 pre-release AppImage from GDrive repository.
>>> >> 2/ start a clean instance of digiKam from scratch.
>>> >> 3/ Import step by step albums. Not all in one step.
>>> >> 4/ When the album containing the image freeze digiKam, share the datat
>>> >> somewhere on the web to test here in local.
>>> >>
>>> >> Note : if there is a crash in the thread due to Exiv2, the crash will
>>> >> not close the application, but we can catch it to identify the
>>> >> problem. Run the AppImage from the console with the "debug" argument.
>>> >> This will run digiKam in GDB. Run AppImage with "help" argument for
>>> >> details.
>>> >>
>>> >> Best
>>> >>
>>> >> Gilles Caulier
>>> >>
>>> >>
>>> >>
>>> >> 2017-06-06 14:44 GMT+02:00 Sigfrid Lundberg
>>> >> <sigfrid at sigfrid-lundberg.se>:
>>> >> >
>>> >> > Dear everybody,
>>> >> >
>>> >> > I'd like first say, warmly, thanks for an excellent product. I've
>>> >> > been a
>>> >> > daily user since about 10 years.
>>> >> >
>>> >> > I've a collection of close to 10000 raw images  *.DNG, *.ORF and
>>> >> > *.RW2
>>> >> > formats. DNG is about 75% of these. I've kept about 10% of the photos
>>> >> > I've
>>> >> > made. digikam has been essential in weeding and curating the
>>> >> > collection
>>> >> > (from about 100000 to 10000), and writing captions, applying
>>> >> > tags/keywords
>>> >> > and geo-tagging the all items.
>>> >> >
>>> >> > I've always ensured that data is written back to the exif header and
>>> >> > that
>>> >> > there are xmp files in the file system.  The collection is stored in
>>> >> > a
>>> >> > file
>>> >> > system which is also a part of my Dropbox area.
>>> >> > I've share the same collection over more than one computer, but then
>>> >> > had
>>> >> > the
>>> >> > sqlite database files in directories outside Dropbox. Worked without
>>> >> > problem.
>>> >> >
>>> >> > A few nights ago, my computer wanted to go from Ubuntu 14.04 LTS to
>>> >> > 16.04
>>> >> > LTS, and I decided to allow that. When doing that, I felt that it
>>> >> > would
>>> >> > be
>>> >> > time to go to digikam 5.5.0. It just didn't work.
>>> >> >
>>> >> > I've
>>> >> >
>>> >> > 1. Installed the appimage (5.5.0 and 5.6.0)
>>> >> > 2. Installed the digikam5 version for ubuntu according to tutorials
>>> >> > on
>>> >> > the
>>> >> > Net.
>>> >> > 3. I've compiled it from sources (been through all the cmake and
>>> >> > dependency
>>> >> > manage stuff), and got executable binaries.
>>> >> >
>>> >> > finally I
>>> >> >
>>> >> > 4. Downgraded to digikam 4 again.
>>> >> >
>>> >> > Sorry, but it doesn't work. There is no difference in having the
>>> >> > database
>>> >> > files inside our outside the collections. Not even downgrading
>>> >> > helped.
>>> >> >
>>> >> > What happens in all four cases is that digikam reads about 25% of the
>>> >> > collection, and then it sighs deeply, printing to STDOUT what I've
>>> >> > added
>>> >> > below (seen people mentioning similar problems on mailing lists, but
>>> >> > I've
>>> >> > not found a clear answer).
>>> >> >
>>> >> > The raw image in question is perfectly readable:
>>> >> >
>>> >> > https://www.dropbox.com/s/zwh1seo5gced12b/R0011071.DNG?dl=0
>>> >> >
>>> >> > Apart from dropbox (which actually make the raw processing without
>>> >> > problems), I have successfully read this image using gimp and
>>> >> > rawtherapee,
>>> >> > so the file shouldn't be corrupt. After failing on that one, the
>>> >> > process
>>> >> > won't  recover. The corresponding xmp file is well formed according
>>> >> > to
>>> >> > xmllint.
>>> >> >
>>> >> > Closing digikam, and doing  dump and refresh of the  database using
>>> >> > sqlite3
>>> >> > command line tool doesn't help.
>>> >> > It still tries to reread the collection and ends at the same item. If
>>> >> > I
>>> >> > move
>>> >> > this image outside the collection, remove the database files and
>>> >> > start
>>> >> > over
>>> >> > it ends the same way for some other item.
>>> >> >
>>> >> > Right now I'm desperate since my work-flow is gone.
>>> >> >
>>> >> > Any clue, anyone?
>>> >> >
>>> >> > Thanks in advance
>>> >> >
>>> >> > Sigfrid
>>> >> >
>>> >> >
>>> >> >
>>> >> >
>>> >> > digikam.metaengine: DateTime (Exif digitalized):  tors dec. 22
>>> >> > 20:37:23
>>> >> > 2011
>>> >> > digikam.metaengine: Orientation => Exif.Image.Orientation =>  1
>>> >> > digikam.database: Scanning took 2 ms
>>> >> > digikam.database: Finishing took 0 ms
>>> >> > digikam.dimg:
>>> >> > "/home/sigge/Dropbox/galleri/201112-201112/R0011071.DNG"
>>> >> > :
>>> >> > RAW file identified
>>> >> > digikam.geoiface: ----
>>> >> > digikam.geoiface: ----
>>> >> > digikam.general: Using  4  CPU core to run threads
>>> >> > digikam.general: Stacked View Mode :  0
>>> >> > digikam.general: Action Thread run  1  new jobs
>>> >> > digikam.dbengine: Database is locked. Waited 0
>>> >> > digikam.geoiface: ----
>>> >> > digikam.dbengine: Database is locked. Waited 250
>>> >> > digikam.dbengine: Database is locked. Waited 500
>>> >> > digikam.dbengine: Database is locked. Waited 750
>>> >> >
>>> >> > ....
>>> >> >
>>> >> > digikam.dbengine: Database is locked. Waited 9500
>>> >> > digikam.dbengine: Database is locked. Waited 9750
>>> >> > digikam.dbengine: Database is locked. Waited 10000
>>> >> > digikam.dbengine: Detected locked database file. There is an active
>>> >> > transaction. Waited but giving up now.
>>> >> > digikam.dbengine: Failure executing query:
>>> >> >  "SELECT id FROM Albums WHERE albumRoot=? AND relativePath=?;"
>>> >> > Error messages: "Unable to fetch row" "database table is locked:
>>> >> > Albums"
>>> >> > 6 1
>>> >> > Bound values:  (QVariant(int, 1), QVariant(QString,
>>> >> > "/200708-200712"))
>>> >> > digikam.general: Data From DBJobsThread is null:  true
>>> >> > digikam.general: Cancel Main Thread
>>> >> > digikam.general: One job is done
>>> >> > ^C
>>> >> >
>>> >> >
>>> >> >
>>> >> >
>>> >> > --
>>> >> > Sigfrid Lundberg, Ph.D., System developer
>>> >> > Lund, Sweden
>>> >> > http://sigfrid-lundberg.se/
>>> >
>>> >
>>> >
>>> >
>>> > --
>>> > Sigfrid Lundberg, Ph.D., System developer
>>> > Lund, Sweden
>>> > http://sigfrid-lundberg.se/
>>
>>
>>
>>
>> --
>> Sigfrid Lundberg, Ph.D., System developer
>> Lund, Sweden
>> http://sigfrid-lundberg.se/



-- 
Sigfrid Lundberg, Ph.D., System developer
Lund, Sweden
http://sigfrid-lundberg.se/