[Kdenlive-devel] [Mlt-devel] video stabilization

[Dan Dennedy]

videostab is now in MLT git. I basically rewrote the MLT
filter-wrapper. By nature, this is a two stage process: analyze and
apply.
Previously, on the first frame it would get the frame's producer, loop
over all of its frames doing the analysis (very slow) and then write
the result (set of vectors) to a text file. Basically, plugins are not
allowed to affect seeking.

My rewrite uses mlt_geometry to serialize and deserialize the vector
set. The first run puts the result into a property named "vectors." I
added an all=1 property to the XML consumer to make it process all of
the input frames before serializing to XML. This makes a MLT XML
containing the videostab.vectors property. On the second run, when the
filter sees the vectors property, it applies the image adjustments.
This technique will be used to add a proper audio normalization
(replay gain determination and application).

Here is an example of melt usage:
pre-process:
$ melt -verbose -profile quarter_ntsc_wide capture001.m2t.mpg out=1000
-filter videostab -consumer xml:capture001-vstab.mlt all=1
real_time=-2

(While the filter is itself not concurrent, real_time=-2 allows
decoding and scaling to be concurrent with videostab analysis.)

preview:
$ melt capture001-vstab.mlt

encode:
$ melt capture001-vstab.mlt  -verbose -consumer
avformat:capture001-vstab.webm properties=webm real_time=-2 threads=3

This does require special integration into Kdenlive, but at least it
will not cause users to think Kdenlive hung or stuck in an infinite
loop. ;-) Kdenlive can already load the output MLT XML as a virtual
clip, but most users will not know how to do the analysis step.
Kdenlive does NOT need to use the XML consumer. It can load the
producer, attach the filter, cycle through the frames, get the vectors
property, and put it in its project. Maybe this require a button on
the effect's panel labelled Analyze or similar. Should this process be
modal or run in the background?

On Sat, Aug 13, 2011 at 2:44 AM, Marco Gittler <marco at gitma.de> wrote:
>
> Am 12.08.2011 um 19:50 schrieb Dan Dennedy:
>
>> On Fri, Aug 12, 2011 at 1:14 AM, Marco Gittler <marco at gitma.de> wrote:
>>>
>>> Am 12.08.2011 um 08:05 schrieb Dan Dennedy:
>>>
>>>> On Thu, Aug 11, 2011 at 10:20 PM, Marco Gittler <marco at gitma.de> wrote:
>>>>>
>>>>> Am 11.08.2011 um 22:54 schrieb Dan Dennedy:
>>>>>
>>>>>> On Thu, Aug 11, 2011 at 10:33 AM, Marco Gittler <marco at gitma.de> wrote:
>>>>>>> Hi
>>>>>>>
>>>>>>> i made an mlt plugin, that uses the stabilization from http://vstab.sourceforge.net/.
>>>>>>> First tests are working very good.
>>>>>>
>>>>>> What a nice contribution!
>>>>> :)
>>>>>>
>>>>>>> There a lots of speed up to be done (cache calculated values in a file, for later usage).
>>>>>>
>>>>>> I have not looked at this yet, but can you simply put the calculated
>>>>>> values in a string property (colon-delimited list of numbers or json
>>>>>> ala rotoscoping). Then, when serializing a mlt composition to xml, it
>>>>>> will be saved. Next, when loading a mlt xml, the plugin can use the
>>>>>> values from the property instead of calculating them?
>>>>> i implemented this now.
>>>>> but it looks first it is not working. but  i found out, the filter will have 2 runs with the video file, started with
>>>>>  melt ~/Desktop/IMG_0286.MOV -filter videostab -verbose
>>>>>
>>>>> on the first run it "deshakes" for 38 frames (and saves the values)
>>>>> on 2nd run (the visible on) there are 45 (correct length) frames.
>>>>> how can the first (false length) run be prevented, else the vector data are not correct ?
>>>>>
>>>>> i  had also a first run with 45 frames (a lucky moment) , then it will work perfect.
>>>>
>>>> It sounds like in the first run, it is frame-dropping if you are using
>>>> the sdl consumer. Try running with '-consumer sdl real_time=-1'. Hmm..
>>>> if you run the first pass with '-consumer xml:test.mlt' to save the
>>>> values then it will not run through all of the frames, so that mode of
>>>> operation will not work as I originally thought. I think we need to
>>>> change the XML consumer to add an option to iterate through each
>>>> frame. This weekend I will download your plugin and figure out what to
>>>> do.
>>>
>>> some more test with other videos are working. it seems that the problem exists only with video @30fps
>>> the first run will then be done with 38 frames (46 at 30 fps= 38 @25 fps with the same time to run)
>>> 25fps videos does not have this "bug".
>>>
>>> so i don't know if this must be fixed, or if this is a "video profile" thing that must be set for this.
>>>
>>> melt -profile ntsc_dvd ~/Desktop/IMG_0286.MOV -filter videostab -verbose and
>>> melt -profile pal_dvd ~/Desktop/IMG_0286.MOV -filter videostab -verbose
>>> the video has every time 38 frame (on the 2 runs)
>>
>> Unless you made those profiles yourself, they do not exist. See 'melt
>> -query profiles' to see a list of what is "installed." Try using
>> dv_ntsc  and dv_pal.
>>
>
> You'r right. The profiles ares not installed, but i wonder now, why the number of frames in the first and second run are different.
> same video ( 30fps short video from phone) , with dv_pal, dv_ntsc and without profile
>
> profile         1st run       2nd run
>
> without         38             46
> dv_pal         38              38
> dv_ntsc       45              45
>
> It looks like, it has something to do with the missing profile.
> For work with kdenlive, this would be no problem, since there will be a profile set up.
> Only for simple test on command line without profile this would harm the result.
>
> regards marco
>
>
>>>
>>> melt ~/Desktop/IMG_0286.MOV -filter videostab -verbose
>>> will have first run 38, second 46 frames. (it is a short video from an iPhone)
>>>
>>> regarde marco
>>>
>>>>
>>>>> regards marco
>>>>>
>>>>>>
>>>>>> --