[Kdenlive-devel] mlt jack issues - episode 3

[Ed Rogalsky]

2012/11/4 Dan Dennedy <ddennedy at gmail.com>:
> On Thu, Oct 4, 2012 at 2:31 AM, Ed Rogalsky <ed.rogalsky at gmail.com> wrote:
>> Hi Dan,
>>
>> the last days I worked on the jack seeking and I hope this is the last
>> major step before contributing the code to kdenlive.
>> There are some problems with the current jack_sync implementation.
>>
>> I parsed the jack documantion and found that it is a little bit
>> unclear. Thats why I studied some jack clients like ardour or
>> qtractor and the jackd implementation. Here the results:
>>
>> Jack Sync theory:
>> ==============
>>
>>     The jack_sync callback is designed only for position relocation
>> (seeking) and slow sync clients (sync callback registered).
>>     The idea is to give the clients enough time for seeking. Indeed
>> "video clients" are very slow in seeking.
>>     Starting jack transport is per defintion a "relocation". So
>> jack_sync is only called (for registered client) if a relocation is
>>     requested or the jack transport is started(ing). The jack_sync is
>> not the transport transition callback: started => stopped and so
>>     on. For getting the transport transition info it has to be polled
>> e.g. from the jack_process by calling the jack_transport_query
>>     function (follow_transport_slave principle). There can be fast
>> clients with no jack_sync callback registered.
>>
>>     The jack transport is started if all clients return 1 from the
>> sync_callback or the registered timeout (here: 5s) elapse. After
>>     returning 1 the sync_callback is not called for the entire client.
>>
>> Back to mlt:
>>
>> Issue 1:      _last_pos property
>> ======
>>
>> The _last_pos property is not updated if we have the following timeline:
>>
>> +---------+----------------------+
>> | blank    | video.mpg            |
>> +---------+----------------------+
>> 6s            e.g. 20s
>>
>> or
>>
>> #sh: MLT_CONSUMER=sdl_preview ./melt -jack -blank 125 video-file.mpg
>>
>> In this case you will notice that the jack transport is started after
>> 5s when the timeout elapse and the jackd starts the transport.
>> The problem is that in the "blank" time the function
>>
>> jackrack_get_audio
>>
>> is not called and we don't get the position update.
>
> This issue is fundamental to the design of MLT, which probably makes
> MLT's native support for transport sync non-universal and not suitable
> for some applications like Kdenlive unless Kdenlive wants to start
> putting silent clips on the timeline instead of blanks. Why? because
> MLT does not ask filters to process "null" data, which is what blank
> represents. Trying to change that is likely to introduce bugs.
>
>> jack_sync:
>>
>>                 else if ( position >= mlt_properties_get_position( properties,
>> "_last_pos" ) - 2 ) => based on the "_last_pos" this criteria is not
>> matched or incorrect
>>                 {
>>                         mlt_properties_set_int( properties, "_sync_guard", 0 );
>>                         result = 1;
>>                 }
>>
>>
>> case 1:
>>
>> not updated "_last_pos" > position => timeout
>
> Getting a non-existent property as a numeric yields a 0, so this case
> is highly unlikely.

Oh I see the problem. What you mean is only true for the first time if
the property is
read and it is not set to a value yet => correct you get 0. But
consider the following situation:

In our timeline (see above):

1) move playhead to timecode 7s (after blank). The _last_pos property
is updated to a value != 0 (timecode = 7s).
2) move playhead back to say 2s. The "_last_pos" property (pos == 7s)
is not updated due to the blank problem.
    Try to start jack playback => jackd timeout.

>>
>> case 2:
>>
>> not updated "_last_pos" < position => jack transport starts, but hay
>> did we wait for mlt seeking???
>
> This is just firing an event making it dependent upon the app's
> behavior, but mlt_producer_seek() is not synchronous (impossible) and
> neither does melt wait for the seeked frame to be reported as
> displayed by the consumer. It may be possible in Shotcut and Kdenlive
> with some work.
>
>>
>>
>> Issue 2: mlt transport is started ignoring slower sync clients
>> ======
>>
>> jack_sync():
>> ....
>>         else if ( state == JackTransportStarting )
>>         {
>>                 result = 0;
>>                 if ( !mlt_properties_get_int( properties, "_sync_guard" ) )
>>                 {
>>                         mlt_properties_set_int( properties, "_sync_guard", 1 );
>>                         mlt_events_fire( properties, "jack-started", &position, NULL );
>>                 }
>>                 else if ( position >= mlt_properties_get_position( properties,
>> "_last_pos" ) - 2 )
>>                 {
>>                         mlt_properties_set_int( properties, "_sync_guard", 0 );
>>                         result = 1;
>>                 }
>>         }
>> ...
>>
>> Considering the fact "_sync_guard == 0" the "jack-started" event is
>> fired and the mlt transport starts. This is a problem if we have
>> some slower sync clients connected. The jackd holds the jack transport
>> in "starting" until all clients return 1 or the timeout elapse.
>> In this case the mlt client gets absolutely out of sync.
>>
>> As mentioned above the mlt transport should be started from the
>> jack_process by polling the jack_transport_query (follow_slave
>> principle).
>> And in the jack_sync only "mlt seeking" should be requested. This is
>> the way it is implemented in ardour.
>>
>> In my private mlt jack  branch I introduced the event "jack-starting"
>> - giving the appl the time for seeking or starting a very complex and
>> slow
>> enginge but not the transport :-))
>>
>> See for inspiration:
>> https://github.com/eddrog/mlt/blob/jack/experimental/src/modules/jackrack/filter_jackrack.c
>>
>> As a work around for "_last_pos" I'm syncing the jack position to
>> consumers position. Sorry for the "jack-last-pos-req" event. I tried
>> to reach the
>> consumers object from the filter. But as result I got just crashes.
>
> Yeah, you cannot really do that. The filter should only fire events,
> and let the application access the consumer.
>
>> In the appl I set the "_last_position" property with the conumers
>> position value.
>>
>> My implementation is very experimental but I get pretty good sync
>> results (seeking while playing).
>>
>>
>> I know you have a better way to solve the issues but I think my way is
>> a good base for inspiration :-)))
>>
>
> I think there are some very fundamental issues with the integration of
> jack transport sync within a MLT filter for all use cases, and it
> should instead be done entirely within Kdenlive.

How should we proceed here? What is your plan / idea:

1) remove jack_sync callback from filter_jackrack and do it in app

or

2) provide some additional events from filter <=> app (similar way I did in the
    experimental branch)


>>
>> Issue 3: sound starts to early about 1s
>> ======
>>
>> +---------+----------------------+
>> | blank    | video.mpg            |
>> +---------+----------------------+
>> 6s            e.g. 20s
>>
>> consumer=sdl_preview
>>
>> We have again our problematic timeline. In this case the sound starts
>> to play ca. 1s to early. I tried to figure out why but hay.... :-))
>>
>>
>>
>>
>> Issue 4: sound plays after stopping (ca. 1s)
>> ======
>>
>> I have still the problem that after stopping you hear the sound
>> playing ca. 1s. It becomes very crazy if you have a transport loop
>> e.g. in ardour and
>> a sound track in kdenlive. After looping some times and stopping it
>> plays and plays and plays (ca. 5s or so). For me this issue is low
>> prio I think.
>>
>>
>>
>>
>> Ok now I'm done - puh.
>>
>> regards
>>
>> eddrog
>
>
>
> --
> +-DRD-+