Issues with Exiv2...

[Gilles Caulier]

With this commit :

https://commits.kde.org/digikam/134d5731cdc5f0e99df3839f943bd60804e96072

I introduced the first stage to consolidate metadata extraction with Exiv2
using multi-threading.

Gilles Caulier

Le jeu. 8 nov. 2018 à 09:22, Gilles Caulier <caulier.gilles at gmail.com> a
écrit :

> Hi all,
>
> With this commit :
>
> https://commits.kde.org/digikam/1b434ace96e4bec1392ae14800e1fbb8a5bb6071
>
> The unit test settings can be customized to perform a check over a local
> collection. I would to know if crash can be reproducibled when metadata are
> read from files in other computer.
>
> Best
>
> Gilles Caulier
>
> Le jeu. 8 nov. 2018 à 00:18, Gilles Caulier <caulier.gilles at gmail.com> a
> écrit :
>
>> Another second big problem with Exiv2 is the memory allocation of
>> DMetadata before to use, especially with large files to parse (depending of
>> file mime types).
>>
>> DMetadata (and in fact the internal Exiv2 objects) need to be allocated
>> on heap, not on stack. Similar problem was discovered by Maik with last
>> libraw 0.19 objects.
>>
>> So the DMetadata wrapper need to use QScopedPointer. See my code from
>> this unit test function :
>>
>>
>> https://cgit.kde.org/digikam.git/tree/core/tests/metadataengine/metareaderthreadtest.cpp#n60
>>
>> Gilles Caulier
>>
>> Le mer. 7 nov. 2018 à 23:41, Gilles Caulier <caulier.gilles at gmail.com> a
>> écrit :
>>
>>> Hi all,
>>>
>>> Actually, i write and test with unit-tests, over Exiv2 and metadata
>>> extraction.
>>>
>>> In DK, the metadata wrapper over Exiv2 is based on MetaEngine and the
>>> inherited class DMetadata. All my tests use DMetadata which provide the
>>> most complete API to play with information.
>>>
>>> I already found few internal problems, and with the help of Maik, we fix
>>> all this stuff.
>>>
>>> But the last unit test to finalize, is the most problematic : the stress
>>> access to Exiv2, with muti-cores and multi-threads.
>>>
>>> Typically, i setup a manager which scan a huge collection of files
>>> (40.000 JPEG, PNG, TIFF, RAW, Video, etc.). Each file is read from
>>> separated thread to catch most important informations (the famous ones
>>> stored in database). Here, 8 cores are used, and 32Gb of ram permit to
>>> eliminate all memory leak side effects.
>>>
>>> The unit test do not play with database, but stress only Exiv2. It's
>>> enough and in fact show quickly the Exiv2 limitations :
>>>
>>> - Exiv2 API are not re-entrant.
>>> - This is not only limited to files parser.
>>>
>>> In digiKam, all is multi-threaded, especially the database scanners, and
>>> the maintenance jobs. The threads running Exiv2 API can crash suddenly and
>>> let's the application in pending states for a while. Depending of the
>>> context in memory, the application can crash or not, but in all cases, the
>>> threads management is broken and application will not responding.
>>>
>>> The only solution in whole digiKam implementation is to wrap all calls
>>> to DMetadata to a thread-safe interface using Mutex Locker. This will
>>> reduce the performances, but gain for stability is incredible.
>>>
>>> Best
>>>
>>> Gilles Caulier
>>>
>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.kde.org/pipermail/digikam-devel/attachments/20181108/2c6e59cd/attachment.html>