Puttin' on the GL(itz)

Tue May 15 13:10:34 UTC 2012

On Sunday, May 06, 2012 04:15:03 PM ext BogDan Vatra wrote:
> Hello everyone,
> 
>   A few days ago I wrote an mail about "Qt Multimedia status on
> Android" [1], because I don't like to give up that easy, I tried to
> find another solution and I think I found one (at least for
> android-14+). Also some people post other solutions, and I'd like to
> thank them very much! Their methods can be used for older android
> versions.
>   At the end of that email, I complained about the fact that there is
> no fast way to read the texture from graphics memory back to the
> system memory.
>   After several days of reflection (and more investigations), it
> striked me another idea: to move everything in video memory, instead
> to read textures from the video memory.
> 
> Before we go further, I'd like to make some clarifications :
>  * in order to understand this mail, first, you must read "Qt
> Multimedia status on Android" [1] !
>  * this is just a draft, the is no code behind it, so, I don't know if
> it's going to work or not, just my thoughts!
>  * my OpenGL silks are near zero, any help will be very appreciated,
> also, if anyone spots any mistakes *please* reply to this mail !
> 
> Currently Qt on Android uses two methods to draw the controls: raster
> or opengl rendering.
> OpenGL rendering is done using your device graphics card, it doesn't
> seem to be the best solution [2],[3] to draw controls, but is good for
> complex animations. It also have a big problem, currently it is
> limited to a single Top Level Widget (I'll use TLW abbreviation from
> now on) because Android OpenGL is limited to a single window surface
> per process. Let's forget about OpenGL for now, we'll come back to it
> latter.
> Raster rendering is done using a QImage for every TLW, is precise and
> fast (if you don't have complex drawings and/or animations). So we
> have a few QImage object which must be converted into gl textures,
> then we can draw them to the windows surface, here I found that there
> are two methods to do it, first one is to use glTex[Sub]Image* (this
> one seems to be a little slow [4] on some devices, so maybe we should
> avoid it ?) and another one is to use
> eglCreateImageKHR/glEGLImageTargetTexture2DOES which seems much
> faster, the problem with the second one is that we must use a few
> non-public APIs to allocate the graphics buffer, even so, if is faster
> than glTex[Sub]Image, 

Does this mean you can get a eglImage sitting on application-space memory 
(aka, QImage) without a copy?

Technically it's possible on a unified-memory system ... i guess ..., I'm just 
wondering because I've never seen any platform which can do that
(Not even a platform which wasn't caring about memory management at all (e.g. 
vxworks))

(fyi: I've used eglCreateImageKHR before ... it's main intention is to bind 
EGL and GL resources ... not software-resources
   ... but who knows ;)

.. but ok ... if that's the case: cool! That's going to have very good 
performance!

But I suspect(!) you'll end up with a copy anyway ... so:
If there is a copy involved, it doesn't really matter where/who/how you copy 
as long as you copy junks (and not pixels) ... and glTexImage is your friend 
here :)

.... it shouldn't be that big of a deal ... it's not helping the raster-
backend-performance ... but if you want to mix with video I guess there is no 
way around it.

> IMHO it should not be a huge problem.
> This is the way I'm planning to draw the raster images using OpenGL,
> but where is Android's multimedia frame texture in this equations ?
> To be able to display that texture I'm planning to add a new surface
> type which will be linked to a TLW, this is the draft for the
> interface:
> 
> class QSurfaceTexture
> {
>    public:
>    virtual void paintTexture() = 0; // here the widget will draw the
> texture by itself.
> }
> 
> and I'm planning to add a few methods to QPlatformWindow, these
> methods will be used to register/unregister a texture surface and to
> iterate the textures.
> class QPlatformWindow
> {
> ....
>   /// this function will be used to link a new surface texture to a TLW
>   /// order is used to specify the drawing order,
>   ///  - negative values (or 0) means the texture will be draw under the TLW
> ///  - positive values means the texture will be draw over the TLW virtual
> void addSurfaceTexture(QSurfaceTexture *surface, int order = 0 ); virtual
> void removeSurfaceTexture(QSurfaceTexture *surface);
> 
>   /// gets the surfaces textures ordered by order value
>   virtual const QList<QSurfaceTexture *> & belowSurfaceTextures();
>   virtual const QList<QSurfaceTexture *> & aboveSurfaceTextures();
> }
> 
> So, how a qt android multimedia widget will look like ?
> class QAndroidMediaSurfaceTextureWidget: public QWidget, public
> QSurfaceTexture, public AndroidSurfaceTexture
> {
> public:
>       QAndroidMediaSurfaceTextureWidget( ... )
>       {
>          platformWindow()->addSurfaceTexture(this, -1); // the texture
> will be draw under the widget
>       }
> 
>      void paintEvent ( QPaintEvent * event )
>      {
>        ...
>         p.fill(qRgba(255,255,255,255)); // the widget must be filled
> with a transparent color, otherwise the texture behind it will not be
> visible.
>      }
> 
>      void paintTexture()
>      {
>          glBindTexture(GL_TEXTURE_2D, androidTextureName);
>          // draw the texture
>      }
> }
> 
> then a simple window manager will paint all TLW on the window surface:
> 
> drawWindows()
> {
>    eglMakeCurrent(...);
>    foreach ( QPlatformWindow * window, m_windows)
>    {
>          foreach (QSurfaceTexture * surfaceTexture,
> window->belowSurfaceTextures())
>                surfaceTexture->paintTexture(); // first draw all below
> textures
> 
>         drawWindow(window); // update texture for that window, than draw it.
> 
>          foreach (QSurfaceTexture * surfaceTexture,
> window->aboveSurfaceTextures())
>                surfaceTexture->paintTexture(); // last draw all below
> textures }
>    eglSwapBuffers(...);
> }
> 
> Now let's go back to the OpenGL rendering issue. Instead to use a
> windows surface, I'm thinking to switch to a pixelbuffer surface
> convert it to a texture (using glCopyTexImage2D ? ) then draw the
> texture to window surface as the other TLWs.

No :) If you use OpenMAX AL you can let it render into a EGLSurface 

And then there are functions to get a texture out of that,
eglCreateImageKHR   <= this is one of the possible friends
(this one would mean you first create a texture, map a EGLSurface around it, 
and give that to openmax, there are other functions for the other way around)

and then all you need to do is to render it using very limited opengl code.

There is already a QtMultimedia-backend using OpenMAX AL, it's for symbian 
tough!... Not sure how much code-reuse is possible but it's definitely a very 
good reference!

qt-mobility/plugins/multimedia/symbian/openmaxal

The awesome part of this is that the only data going over the GPU-bus is the 
encoded(!) file/data   (aka. zero-copy) 
.... and the GPU-bus is one of the weak links on all platforms.

For the fist case: raster:
If you go to way to copy/map the QImage's to EGL/GL resources, then 99% of all 
openmax al code should be reusable.

> 
> Any comments, suggestions and *help* will be very appreciated.
> 
> Cheers,
> BogDan.
> 
> 
> [1] http://mail.kde.org/pipermail/necessitas-devel/2012-May/000900.html
> [2] http://code.google.com/p/android-lighthouse/issues/detail?id=4#c28
> [3] http://code.google.com/p/android-lighthouse/issues/detail?id=4#c23
> [4]
> http://groups.google.com/group/android-developers/browse_thread/thread/840e
> 4e20faab5c31 ,
> https://developer.qualcomm.com/forum/qdevnet-forums/mobile-gaming-graphics-
> optimization-adreno/1969 _______________________________________________
> Necessitas-devel mailing list
> Necessitas-devel at kde.org
> https://mail.kde.org/mailman/listinfo/necessitas-devel