FFMPEG video metadata parser into 6.0.0

[Andrey Goreev]

Gilles
Thank you for your hard work on this.
I think most of video editing software uses QuickTime tags. See table 3-6 on this webpage:https://developer.apple.com/library/content/documentation/QuickTime/QTFF/Metadata/Metadata.html
I am wondering if you are going to check if file managers display the metadata written by digikam.Dolphin's Info panel does show title and tags written in some of my video files. So does Windows File Manager. I am not sure about Apple OS file manager since I have never used it.
Also you are correct, ffmpeg always has an input file (original) and an output file (result). Audio and video streams can be copied but I bet this operation will introduce some latency. So I would make the write support for video files optional and I would make sure it writes only after an input from user (similar to how the lazy synchronization works)
Cheers,Andrey

Sent from my Samsung Galaxy smartphone.
-------- Original message --------From: Gilles Caulier <caulier.gilles at gmail.com> Date: 2018-03-01  4:19 AM  (GMT-07:00) To: "digiKam developers room - digiKam: Manage your photographs as a professional with the power of open source" <digikam-devel at kde.org> Subject: FFMPEG video metadata parser into 6.0.0 
Hi all,
As you seen certainly, after many years to wait a change from Exiv2 to support video metadata safety, we lost certainly a chance to see this Exiv2 code reviving in the future.
THE STATE : 
- Exiv2 video support is bugous and crash immediately digiKam if collections has video files.- i rerouted more than 60 bugs about Exiv2 video crash. There are certainly more files bugs it's a very long task to analyze GDB backtrace, when it's available.- Since 5.3.0, all bundles drop Exiv2 video support to turn off the cmake option at compilation time.- Even if video support is disabled in Exiv2, only the video file parser are not compiled. All XMP multimedia XMP namespace still here and can be used in RW.
THE SOLUTION :
- As we use QtAv for playing with video, FFMPEG is used in background.- QtAv do not provide a simple wrapper to extract video metadata, even if these info can be displayed in file properties from QtAV player. I reported this whish in QtAV bugzilla, but lead developer estimate this change as out of scope of the project.- I add FFMPEG dependencies in digiKam (libav C API). This is not a really new dependency as we use it indirectly through QtAV. In all case to compile bundles, we need FFMPEG API installed. So the puzzle is not increased...- I write a good FFMPEG metadata parser with a Exif/IPTC/XMP wrapper. The code currently start ffmpeg API (as ffprobe do from CLI) and look for metadata entry from each streams (main container, video container, audio container)
https://cgit.kde.org/digikam.git/tree/libs/dmetadata/dmetadata_ffmpeg.cpp?h=development/6.0.0

- The parser is voluntary limited to one video stream and one audio stream. The standard XMP Multimedia namespaces do not support multiple streams, as i seen (fix me if i'm wrong here). So while all streams from a video container are parsed, when first audio and video entry is found, there are used to populate XMP. Other entries are ignored.
The parser will fix 95% of use cases to populate the Database entries. The goal is to be able to found video by date, make/model, orientation, codecs, size, etc... And the best, GPS info are also extracted if information exists in file. So a search over a map will be possible...
But there is more : internal ffmpeg tags parser is able to get some entries set by video production softwares, as comments, rating, keywords, categories, etc. All these information are dispatched everywhere into Exif, Iptc and Xmp for the pleasure of digiKam database file scanner. 
Important : the parser if very fast, as no video data is decoded to extract metadata. I write 4 CLI unit test tools to show lead info, all Exif, all Iptc, and all Xmp metadata from CLI (these tools work also with images). Look loadfromfile_*.cpp files here :
https://cgit.kde.org/digikam.git/tree/tests/dmetadata?h=development/6.0.0

THE FUTURE:
Currently, we have a code to read only the video metadata. What about writting support ?
I found this kind of code in GitHub :
https://github.com/kritzikratzi/ofxAvCodec/blob/master/src/ofxAvUtils.cpp#L61

This C++ code is a tool to read metadata (poor implementation compared to mine), and to write. This last part is interesting, and a little bit complex, and i must admit that i'm not sure to understand well how it work exactly. I suspect that a new video file is created from an original one where metadata need to be patched. Stream are backported, and later all metadata rewritten and mixed from scratch. This can be a time consuming and not an optimized way, but at least, it must work. FFMPEG do not permit to play directly with metadata entries, it's not a framework dedicated for that. 
Somebody can help me to decode this method in details, at least to see if my analyze is not too bad ? Thanks in advance.
LIMITATION:
The current internal FFMPEG video parsers are limited. This depend of video format, and metadata tags supported. The best video container supported is MP4 / QuickTime. If you want to see this state improved in the future, please report the problem  to FFMPEG team who maintain this code. Comparing FFMPEG code to ExifTool, and also Exiv2 parsers, i see that FFMPEG do not support yet all entries.
TESTING:
To check at least a large amount of use cases, i will need test video files tagged with video production tools, to see if the digiKam FFMPEG parser do the job properly. Using Web cloud services will be the best way to share files.
Note : here i can test with Final Cut Pro as my son use this software for video editing. I need Vegas Pro, Premiere, and other test items...
Thanks in advance for your feedback 
Gilles Caulier
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.kde.org/pipermail/digikam-devel/attachments/20180301/1275b2b3/attachment.html>