KDE4 and video acquisition

[Laurent Pinchart]

Hi Matthias,

> Phonon is supposed to provide a simple and "task oriented" API to the KDE
> developer. With "task oriented" I mean that the API tries to reflect what a
> KDE program will need and not how the hardware works. So on the one hand
> the knowledge about what the hw can do and what interfaces there are is
> usefull, but on the other hand when designing the Phonon API we should
> concentrate on "use cases" of the API and how a developer would be easily
> able to achieve his goal.

Giving an easy and task oriented API to the user is a double-edged knife. On 
one hand it makes it easy for developers to create applications. On the other 
hand, as the API will not expose advanced features, things will have to work 
perfectly out of the box. The developer won't be able to work around 
limitations or bugs. That being said, I think it's not a bad approach, but 
puts more weigh on the Phonon developers' shoulders.

> For the matter at hand I could think of a "restricted" API (restricted in
> the sense that the developer cannot use or set every feature the hw
> provides) that allows a KDE application to select a video capture source,
> an audio capture source (if both are selected that implicitly means they
> need to be synchronized) and set a few parameters like brightness,
> contrast, hue, saturation (which are usefull to expose to the user of the
> application). Everything else can be configured in a KControl module.
[snip scenario]

The scenario is quite clear and looks interesting. We will have to add a few 
other parameters (pan/tilt/zoom for instance) to the API. What happens if 
tomorrow some hardware implements a new control that is useful in end-user 
applications ? Will it require Phonon support, or could we provide an 
extensible way to access controls without requiring a KDE update ?

> If you agree with me on that the question is still open what the API of
> Phonon needs to expose. The rules for me here are:
> - keep it simple (a developer should be able to use the API by looking at
> the class declaration - no docs, no explanations except the class name and
> method names)
> - make common tasks simple
> - make not so common tasks possible

That's very important.

> - leave as much of the hard work to the backends as possible :-P

Speaking of backends, things are quite clear for sound (NMM, MAS, 
GStreamer, ...), but what about video capture ? Are the required features 
implemented in the backends we will use ? My goal is to have KDE4 support 
webcams out of the box. I'm beginning to understand that it will require much 
more work than I thought if I need to contact developers of all backends and 
convince them that they need to change their API to expose required features. 
The future doesn't currently look very bright to me :-(

> > Audio/video sync will probably be one of the major problems. Even if the
> > device can capture both audio and video, the two streams can be unlocked,
> > and we must provide a way to resynchronize them.
>
> Phonon does not really care. ;-) Well, what Phonon does is to tell the
> backend what the developer wants. The backend then has to take care of
> sync. For example I create an AvCapture object and set the AudioSource to
> the soundcard and the VideoSource to a webcam. The VideoPath and the
> AudioPath are both connected to an AvWriter object:
[snip]

I don't have enough experience with audio/video synchronisation to make 
constructive comments, but don't underestimate the difficulty. For instance, 
UVC webcams expose 2 clocks to the computer (internal device clock, used for 
video capture, and USB bus clock), which must then use the information to 
compute the jitter between the device clock and the computer internal clock. 
The same operation is done for audio, and the audio and video must be 
*processed* (this means modifying the audio and/or video rate by dropping 
frames or changing the audio speed using digital filtering). You can put the 
responsability of those operations on the backend, but make sure you 
understand the issue before designing the Phonon API. Once again, it's a 
double-edged knife. If the audio and video are not synchronized, developers 
will blame Phonon.

Sorry to sound so negative, but I think those points must be well understood, 
otherwise Phonon will follow the path that arts took a few years ago.

Laurent Pinchart