Issues with Exiv2...

[Gilles Caulier]

Another second big problem with Exiv2 is the memory allocation of DMetadata
before to use, especially with large files to parse (depending of file mime
types).

DMetadata (and in fact the internal Exiv2 objects) need to be allocated on
heap, not on stack. Similar problem was discovered by Maik with last libraw
0.19 objects.

So the DMetadata wrapper need to use QScopedPointer. See my code from this
unit test function :

https://cgit.kde.org/digikam.git/tree/core/tests/metadataengine/metareaderthreadtest.cpp#n60

Gilles Caulier

Le mer. 7 nov. 2018 à 23:41, Gilles Caulier <caulier.gilles at gmail.com> a
écrit :

> Hi all,
>
> Actually, i write and test with unit-tests, over Exiv2 and metadata
> extraction.
>
> In DK, the metadata wrapper over Exiv2 is based on MetaEngine and the
> inherited class DMetadata. All my tests use DMetadata which provide the
> most complete API to play with information.
>
> I already found few internal problems, and with the help of Maik, we fix
> all this stuff.
>
> But the last unit test to finalize, is the most problematic : the stress
> access to Exiv2, with muti-cores and multi-threads.
>
> Typically, i setup a manager which scan a huge collection of files (40.000
> JPEG, PNG, TIFF, RAW, Video, etc.). Each file is read from separated thread
> to catch most important informations (the famous ones stored in database).
> Here, 8 cores are used, and 32Gb of ram permit to eliminate all memory leak
> side effects.
>
> The unit test do not play with database, but stress only Exiv2. It's
> enough and in fact show quickly the Exiv2 limitations :
>
> - Exiv2 API are not re-entrant.
> - This is not only limited to files parser.
>
> In digiKam, all is multi-threaded, especially the database scanners, and
> the maintenance jobs. The threads running Exiv2 API can crash suddenly and
> let's the application in pending states for a while. Depending of the
> context in memory, the application can crash or not, but in all cases, the
> threads management is broken and application will not responding.
>
> The only solution in whole digiKam implementation is to wrap all calls to
> DMetadata to a thread-safe interface using Mutex Locker. This will reduce
> the performances, but gain for stability is incredible.
>
> Best
>
> Gilles Caulier
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.kde.org/pipermail/digikam-devel/attachments/20181108/ac8605d1/attachment.html>