make_it_cool: kdelibs/kdemm

[Ryan Gammon]

Matthias Kretz wrote:

>Hi Ryan,
>
>thanks for giving feedback. OK, first to check that I know what you're talking 
>about I try to summarize (the further you get with reading the more I start 
>asking questions rather than telling you what I understood)
>  
>

Sorry 'bout that :-)

>- you're currently working on extending the Helix DNA Client so that media 
>players using it support alternative media engines
>  
>

Close... I'm not planning to do this at the client level, though. I'm 
creating something new that will be used in parallel with the helix 
client within the Helix Player and RealPlayer.


>- you want to have that functionality because the Helix DNA Client media code 
>might not always be the best code to handle some specific format (or it 
>doesn't know how to handle the media format at all)
>  
>

Right.

>- you want to achieve that by starting dbus services that register as media 
>engine and that can then be instructed by Helix to play some certain files 
>(or will you pipe the whole media data over dbus, or URLs and hope the media 
>engine can handle it?)
>  
>

I want to define dbus interfaces that define a baseline set of media 
engine functionality.

People would create dbus services that implement such functionality 
using their media engine of choice.

A player wanting to take advantage of this would look on the dbus for 
media player interfaces and generally send commands like

"play this http://url.com in socket_id XEmbed socket"

... where http://url.com is something the user has entered in the player 
somehow, and where socket_id is a video area (if we're playing back 
video). If the player is audio-only, no XEmbed required.

>- I'm not entirely sure what you want to do with the XEmbed stuff: do you want 
>the "media engines" to show some widget inside the media player? If yes, what 
>widgets? Video window? effects dialog? complete media player GUI?
>  
>

Video window.

>- When talking about permissions for the media engines, does that mean they 
>might run as root (or some user with higher permissions). Doesn't that create 
>security problems when you can "remote control" those services over dbus?
>  
>

Services would run as a non-trusted user, but exploits that allow an 
attacker to get non-trusted access can often do significant damage -- we 
don't really differentiate between root/non-root exploits here, they're 
both bad.

I believe SELinux lets you characterize what a given process should be 
doing. Eg, maybe you can give it paths you can write to, ports it can 
open, etc. This means that even if an attacker does find a security 
exploit, he/she is curtailed in the damage he/she can actually do using 
a SELinux security policy. http://enforcer.sourceforge.net/ is an 
interesting project in this vein.

As media playback engines are
- complicated
- sometimes contain binary-only components which are difficult to audit
- talk to the untrusted internet

... there is an interest in making sure we play well with the security 
stuff.

>- How do you want to make it integrate "well" into the application? Say, I 
>have a media player that uses Helix DNA, now I want to play a media file 
>that's only supported by an external media engine. Do I only get to use a 
>subset of the Helix API (not knowing the API at all I have no clue what I'm 
>talking about ;-) ) or how do you map media player functionality from the 
>media engine to your API?
>  
>

In Helix Player, which is for better or worse GObject-based, I'm going 
to take the dbus interfaces and translate them into GInterfaces.

I'm going to have a GObject implements the GInterface using the 
corresponding dbus calls.

I'm then going to take our current GObject Helix Player based widget, 
and implement the same GInterface using it.

In the code, I'll use the GInterface where possible. Where there is 
helix-only functionality, I'll do a glib-style:
if(HX_IS_PLAYER(obj))
{
    /* do helix-only functionality here */
}


So, say I had the following dbus interface:

  <interface name="org.helix.MediaEngine.Player">
    <annotation name="org.freedesktop.DBus.GLib.CSymbol" 
value="hx_ffmpeg_player"/>

    <method name="Play">
      <annotation name="org.freedesktop.DBus.GLib.CSymbol" 
value="hx_ffmpeg_player_play"/>
    </method>
  </interface>

Where ffmpeg here is compiled for free formats not currently supported 
by the player (tremor & matroska for example).

I would turn this into a GInterface definition, say HXBasicPlayback, and 
implement a GObject that implements HXBasicPlayer, and which goes 
looking on the bus for a org.helix.MediaEngine.Player to talk to. Call 
this GObject HXDBusPlayer.

HXPlayer (in-process) would derive from HXBasicPlayback
HXDBusPlayer (out of process, dbus) would derive from HXBasicPlayback

code would look something like:

HXBasicPlayback* helix_playback = HX_BASIC_PLAYBACK(helix_object);
HXBasicPlayback* dbus_playback = HX_BASIC_PLAYBACK(dbus_object);
...

void func (HXBasicPlayback* playback)
{
    if(HX_IS_PLAYER(playback))
    {
       HXPlayer* player = HX_PLAYER(playback);
      
       /* Statistics may be helix-only */
       hx_player_get_statistic("PacketsLost", &adsf);
    }
}

>- One big disadvantage with out of process media handling has always been 
>debugging. This has hit us pretty hard with MCOP/aRts, but I have to admit 
>with MCOP this was probably worse than it can get with sending instructions 
>over dbus.
>  
>

True, but one should typically be able to gdb attach to a media service.

>Yes. Though I'm not sure if it's really similar enough. The main goal of the 
>kdemm API is, after all, to have a simply to use Qt/KDE style API to do all 
>your common and some not so common media tasks.
>  
>

So far so good :)

>AFAIU you already have that API (not Qt/KDE style, though) and want to extend 
>the implentation behind it to use other engines.
>  
>

Fundamentally, I'm working on a D-Bus API which could have Qt-style 
bindings.

>For kdemm we want to support more than only one engine because the 
>functionality we ask for in our API should be available with most media 
>frameworks and because some users have "special needs". I think this is 
>especially true if you want to make KDE a good platform for pro audio tools 
>that does not get in the way like it currently does with aRts. You rather 
>want it to integrate. (but I guess I'm getting a little off topic here)
>  
>

Not so much integrate as abstract common functionality.

One could, eg, still write a pro-audio app to use something like jack 
directly.

>>I was thinking of leaving the configuring of media engines to
>>distribution integrators & advanced users, punting on the creation of an
>>easy admin-configurable unified media engine configuration interface for
>>now.
>>    
>>
>
>What would be the difference between your "system integrator" and my "KDE 
>admin"? Is it just another name or does he also have another "user task"?
>  
>

A system integrator is more likely to go around tweaking build settings 
("--with-polyaudio" or whatever), editing config files, etc.

An admin would be looking for an end-user configuration tool.

>>I was thinking of leaving this unspecified in what I'm doing... If the
>>user has a sound server installed that can do this, great.
>>    
>>
>
>Yes, but kdemm is about a DE, so we want the desktop experience to be 
>"perfect" ;-) I found that changing the audio volume of some specific program 
>is a rather common task for me - it's often just too complicated to do, so 
>that I won't bother, but if I could...
>How to achieve that without a specific sound server? If the soundserver cannot 
>do it, or if the user doesn't use a sound server at all, this can be done 
>using IPC. The kdemm prototype already has this functionality even though a 
>simple way to access all those volume controls is missing. And it's rather 
>cool. Of course this will only work for apps using kdemm.
>  
>

Right... I guess another way of doing this would be to create a sound 
server with a get_volume / set_volume kind of api that only affects a 
single pcm stream, and maybe a get_master_volume / set_master_volume api.

I'd add support for such a sound server all linux multimedia engines, 
and convince linux distributions to use my sound server.

Linux distributions shipping such a sound server would find that their 
apps had independant volume controls.

>>(on domain constraints)
>>
>>    It must be possible to run the desktop over the network, not only
>>    the GUI, but also audio output. If it is possible, the system should
>>    try to play the audio on the computer where the GUI is shown by
>>default.
>>
>>I was also planning to leave this unspecified, as something sound server
>>dependant.
>>    
>>
>
>Right. This is something the sound server does. What this domain constraint is 
>about is to tell us, that somehow it needs to be possible to do it. If that 
>somehow means the admin has to configure audio output to a sound server than 
>so be it. If it would not be possible at all to forward the audio stream to 
>another computer I'd say kdemm has failed. So much for the rationale to put 
>this constraint in there...
>  
>

Ok, makes sense.


>>(on create reusable audio path)
>>
>>In general, I wasn't planning on assuming a graph-based architecture in
>>the underlying media engine, and in fact was going to punt on effects
>>for now (like eq, reverb, equalizer).
>>
>>I'd probably implement this down the road by creating a basic effects
>>interface that actually has methods to configure cross fading, eq,
>>reverb, etc. rather than doing pipleline manipulation. I think this is
>>going to be tough to do in a cross-media engine way...
>>    
>>
>
>I don't want to assume a graph-based architecture either. That's why I didn't 
>model a flow graph. All I did was put some mysterious "path" in there. How 
>that maps to the media framework should not matter.
>
>What is really more complicated is to map effects from the media framework to 
>interfaces and GUIs. I'm currently working on this.
>  
>

Right -- sounds neat, lmk how it goes :) It would be cool to have 
something like this for sure.


>>Is there any way that the work I'm doing could be useful to you guys?
>>    
>>
>
>I'm not sure. I guess one day kdemm would like to have a Helix DNA Client 
>backend. But this does not have anything to do with the current discussion...
>
>I would be very interested, though, in how exactly you're planning to use 
>dbus. Whether you want to only send instructions are whether you want to send 
>media data over the wire (I guess this is not a task dbus was meant for). And 
>how much communication you think you need to fully integrate the external 
>media engine into your API.
>  
>

We would just send instructions over the wire, not the media data itself.

Communication would be along the lines of:
openUrl("http://www.someurl.com")
play()
stop()

... that kind of thing.


>The interesting part about your approach is that you can use several media 
>engines at the same time, while kdemm in it's current design would always be 
>fixed to exactly one media framework.
>  
>

cool

>Sorry for the long email, I hope I'm making sense (after all, it's been a long 
>day and time to go to bed).
>  
>

Heh, yeah, you're making more sense than I am :)

-- 
Ryan Gammon
rgammon at real.com
Developer for Helix Player
https://player.helixcommunity.org