[GSoC idea] QML/QGraphicsScene Video Painting

[Harald Sitter]

Hello everyone,

My name is Harald, I am student in Austria and I'd like to pick up on
the Phonon idea "QML/QGraphicsScene Video Painting" [1] for GSoC 2011
;)
I have been student for Ubuntu last year (also a KDE related project)
and would like to work on KDE technology this year. I have a bit of
experience with C++/Qt, GStreamer, libvlc and QML, didn't do any GL
stuff yet.

I think this is a very important project as QML is becoming more
important and creating good looking video players is not very easy
with QWidgets and the usually used overlay painting for videos.
Additionally with work being done on the Plasma Media Center a whole
new use case arises that deserves good support for the sake of user
experience (unlike what QGraphicsProxyWidget provides).

I have been looking into this topic quite a bit and noted the
following requirements:
a) must be as fast as possible, no one likes choppy video playback
b) must use as little CPU as possible, since QML is a target on
c) must be maintainable ;)
d) must work everywhere

If you have any further thoughts, I really appreciate input :)

>From above requirements I concluded the following:
Regarding a) and b): painting using opengl should be preferred as
doing it using QImage and friends is rather heavy on the CPU.
Regarding c): as QML (at least currently) is based on the
QGraphicsScene magic an implementation should be done for a
QGraphicsItem (or rather QGraphicsObject) and then just wrapped by a
QDeclarativeItem.
Regarding d): at least GL and raster painting should be implemented,
most mobile devices will have half-way decent GL support, so we can
compensate for the (usually) weaker CPU. On a desktop we have stronger
CPU, so even without GL it should be possible to provide sensible user
experience. Latter will require field tests though, as I am not
entirely sure running a video at 1600xSomething fullscreen will still
work all that well.

Painting in a QGraphicsItem is done by overloading the pure virtual
paint() function, which among other things takes a QPainter as
parameter. This helps a great deal with supporting multiple painting
methods (GL, raster, whatnot). Please have a look at [2] for the
general idea.
We'd have the QML element which is really based on the QGraphicsItem.
The QGraphicsItem uses a (private) QPainter implementation that will
use a well defined interface to get the actual painters to paint
stuff. The painters would be implemented in the backend.

An alternative approach would be to enhance the existing experimental
VideoDataOutput class to work fast enough for live playback, in which
case we would have the painters in libphonon and grab data via the
enhanced interface for VideoDataOutput.

Both options have advantages and disadvantages, while using
VideoDataOutput would make supporting the new QGraphicsItem very cheap
from a backend POV (as both GStreamer and the VLC backend have sort of
working implementations already), it withholds the backend from
further optimizing the painting in what ever ways the underlying
libraries might support. For example I could imagine that in
phonon-gstreamer we could have a gstreamer plugin directly in the
gstreamer pipeline and have it act as painter.
Having the painters in the backends might however lead to duplicated
code (unless I find a way to do the general stuff really nifty in an
abstract base class in libphonon).

Of course there is no reason why we should not do both, surely
VideoDataOutput could use some improvements anyway. Therefore I'd like
to propose starting off with having the painters in libphonon (private
of course) so we get a working solution ASAP and then move on to
providing the backends with the ability to have the painters built in
(if that should be necessary/desired at all).

What do you think?

[1] http://community.kde.org/GSoC/2011/Ideas#Project:_QML.2FQGraphicsScene_Video_Painting
[2] http://i.imgur.com/L8vyr.png

best regards,
Harald