Issues with Exiv2...

[Gilles Caulier]

Hi all,

With this commit :

https://commits.kde.org/digikam/1b434ace96e4bec1392ae14800e1fbb8a5bb6071

The unit test settings can be customized to perform a check over a local
collection. I would to know if crash can be reproducibled when metadata are
read from files in other computer.

Best

Gilles Caulier

Le jeu. 8 nov. 2018 à 00:18, Gilles Caulier <caulier.gilles at gmail.com> a
écrit :

> Another second big problem with Exiv2 is the memory allocation of
> DMetadata before to use, especially with large files to parse (depending of
> file mime types).
>
> DMetadata (and in fact the internal Exiv2 objects) need to be allocated on
> heap, not on stack. Similar problem was discovered by Maik with last libraw
> 0.19 objects.
>
> So the DMetadata wrapper need to use QScopedPointer. See my code from this
> unit test function :
>
>
> https://cgit.kde.org/digikam.git/tree/core/tests/metadataengine/metareaderthreadtest.cpp#n60
>
> Gilles Caulier
>
> Le mer. 7 nov. 2018 à 23:41, Gilles Caulier <caulier.gilles at gmail.com> a
> écrit :
>
>> Hi all,
>>
>> Actually, i write and test with unit-tests, over Exiv2 and metadata
>> extraction.
>>
>> In DK, the metadata wrapper over Exiv2 is based on MetaEngine and the
>> inherited class DMetadata. All my tests use DMetadata which provide the
>> most complete API to play with information.
>>
>> I already found few internal problems, and with the help of Maik, we fix
>> all this stuff.
>>
>> But the last unit test to finalize, is the most problematic : the stress
>> access to Exiv2, with muti-cores and multi-threads.
>>
>> Typically, i setup a manager which scan a huge collection of files
>> (40.000 JPEG, PNG, TIFF, RAW, Video, etc.). Each file is read from
>> separated thread to catch most important informations (the famous ones
>> stored in database). Here, 8 cores are used, and 32Gb of ram permit to
>> eliminate all memory leak side effects.
>>
>> The unit test do not play with database, but stress only Exiv2. It's
>> enough and in fact show quickly the Exiv2 limitations :
>>
>> - Exiv2 API are not re-entrant.
>> - This is not only limited to files parser.
>>
>> In digiKam, all is multi-threaded, especially the database scanners, and
>> the maintenance jobs. The threads running Exiv2 API can crash suddenly and
>> let's the application in pending states for a while. Depending of the
>> context in memory, the application can crash or not, but in all cases, the
>> threads management is broken and application will not responding.
>>
>> The only solution in whole digiKam implementation is to wrap all calls to
>> DMetadata to a thread-safe interface using Mutex Locker. This will reduce
>> the performances, but gain for stability is incredible.
>>
>> Best
>>
>> Gilles Caulier
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.kde.org/pipermail/digikam-devel/attachments/20181108/ae43ea1c/attachment.html>