KDE4 and video acquisition

[Marco Lohse]

Laurent Pinchart wrote:
[..]
>>- leave as much of the hard work to the backends as possible :-P
> 
> 
> Speaking of backends, things are quite clear for sound (NMM, MAS, 
> GStreamer, ...), but what about video capture ? Are the required features 
> implemented in the backends we will use ? My goal is to have KDE4 support 
> webcams out of the box. I'm beginning to understand that it will require much 
> more work than I thought if I need to contact developers of all backends and 
> convince them that they need to change their API to expose required features. 
> The future doesn't currently look very bright to me :-(
> 
> 
>>>Audio/video sync will probably be one of the major problems. Even if the
>>>device can capture both audio and video, the two streams can be unlocked,
>>>and we must provide a way to resynchronize them.
>>
>>Phonon does not really care. ;-) Well, what Phonon does is to tell the
>>backend what the developer wants. The backend then has to take care of
>>sync. For example I create an AvCapture object and set the AudioSource to
>>the soundcard and the VideoSource to a webcam. The VideoPath and the
>>AudioPath are both connected to an AvWriter object:
> 
> [snip]
> 
> I don't have enough experience with audio/video synchronisation to make 
> constructive comments, but don't underestimate the difficulty. For instance, 
> UVC webcams expose 2 clocks to the computer (internal device clock, used for 
> video capture, and USB bus clock), which must then use the information to 
> compute the jitter between the device clock and the computer internal clock. 
> The same operation is done for audio, and the audio and video must be 
> *processed* (this means modifying the audio and/or video rate by dropping 
> frames or changing the audio speed using digital filtering). You can put the 
> responsability of those operations on the backend, but make sure you 
> understand the issue before designing the Phonon API. Once again, it's a 
> double-edged knife. If the audio and video are not synchronized, developers 
> will blame Phonon.
> 
> Sorry to sound so negative, but I think those points must be well understood, 
> otherwise Phonon will follow the path that arts took a few years ago.
> 

NMM http://www.networkmultimedia.org does support synchronized capturing 
from different devices (e.g. a V4L(2) frame grabber/camera and a sound 
board). In fact, this feature was introduced in 2002: 
http://graphics.cs.uni-sb.de/VCORE/

Cheers, Marco