Issues with Exiv2...

[Gilles Caulier]

Hi all,

Actually, i write and test with unit-tests, over Exiv2 and metadata
extraction.

In DK, the metadata wrapper over Exiv2 is based on MetaEngine and the
inherited class DMetadata. All my tests use DMetadata which provide the
most complete API to play with information.

I already found few internal problems, and with the help of Maik, we fix
all this stuff.

But the last unit test to finalize, is the most problematic : the stress
access to Exiv2, with muti-cores and multi-threads.

Typically, i setup a manager which scan a huge collection of files (40.000
JPEG, PNG, TIFF, RAW, Video, etc.). Each file is read from separated thread
to catch most important informations (the famous ones stored in database).
Here, 8 cores are used, and 32Gb of ram permit to eliminate all memory leak
side effects.

The unit test do not play with database, but stress only Exiv2. It's enough
and in fact show quickly the Exiv2 limitations :

- Exiv2 API are not re-entrant.
- This is not only limited to files parser.

In digiKam, all is multi-threaded, especially the database scanners, and
the maintenance jobs. The threads running Exiv2 API can crash suddenly and
let's the application in pending states for a while. Depending of the
context in memory, the application can crash or not, but in all cases, the
threads management is broken and application will not responding.

The only solution in whole digiKam implementation is to wrap all calls to
DMetadata to a thread-safe interface using Mutex Locker. This will reduce
the performances, but gain for stability is incredible.

Best

Gilles Caulier
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.kde.org/pipermail/digikam-devel/attachments/20181107/5d843864/attachment.html>