Puttin' on the GL(itz)

Tue May 15 14:30:15 UTC 2012

Hi Thomas,

> On Sunday, May 06, 2012 04:15:03 PM ext BogDan Vatra wrote:
>>  Hello everyone,
>> 
>>    A few days ago I wrote an mail about "Qt Multimedia status on
>>  Android" [1], because I don't like to give up that easy, I tried 
> to
>>  find another solution and I think I found one (at least for
>>  android-14+). Also some people post other solutions, and I'd like to
>>  thank them very much! Their methods can be used for older android
>>  versions.
>>    At the end of that email, I complained about the fact that there is
>>  no fast way to read the texture from graphics memory back to the
>>  system memory.
>>    After several days of reflection (and more investigations), it
>>  striked me another idea: to move everything in video memory, instead
>>  to read textures from the video memory.
>> 
>>  Before we go further, I'd like to make some clarifications :
>>   * in order to understand this mail, first, you must read "Qt
>>  Multimedia status on Android" [1] !
>>   * this is just a draft, the is no code behind it, so, I don't know if
>>  it's going to work or not, just my thoughts!
>>   * my OpenGL silks are near zero, any help will be very appreciated,
>>  also, if anyone spots any mistakes *please* reply to this mail !
>> 
>>  Currently Qt on Android uses two methods to draw the controls: raster
>>  or opengl rendering.
>>  OpenGL rendering is done using your device graphics card, it doesn't
>>  seem to be the best solution [2],[3] to draw controls, but is good for
>>  complex animations. It also have a big problem, currently it is
>>  limited to a single Top Level Widget (I'll use TLW abbreviation from
>>  now on) because Android OpenGL is limited to a single window surface
>>  per process. Let's forget about OpenGL for now, we'll come back to 
> it
>>  latter.
>>  Raster rendering is done using a QImage for every TLW, is precise and
>>  fast (if you don't have complex drawings and/or animations). So we
>>  have a few QImage object which must be converted into gl textures,
>>  then we can draw them to the windows surface, here I found that there
>>  are two methods to do it, first one is to use glTex[Sub]Image* (this
>>  one seems to be a little slow [4] on some devices, so maybe we should
>>  avoid it ?) and another one is to use
>>  eglCreateImageKHR/glEGLImageTargetTexture2DOES which seems much
>>  faster, the problem with the second one is that we must use a few
>>  non-public APIs to allocate the graphics buffer, even so, if is faster
>>  than glTex[Sub]Image, 
> 
> 
> Does this mean you can get a eglImage sitting on application-space memory 
> (aka, QImage) without a copy?
> 

No, you can't use QImage, you need to create a special graphics buffer
http://androidxref.com/source/xref/frameworks/base/include/ui/GraphicBuffer.h
But you can create a QImage which shares the same memory. 

>
> Technically it's possible on a unified-memory system ... i guess ..., 
> I'm just 
> wondering because I've never seen any platform which can do that
> (Not even a platform which wasn't caring about memory management at all 
> (e.g. 
> vxworks))
> 
> 
> (fyi: I've used eglCreateImageKHR before ... it's main intention is to 
> bind 
> EGL and GL resources ... not software-resources
>    ... but who knows ;)
> 

I don't know if the same memory is also used by the graphics card, but this 
is the way android is doing with video and camera frames
Here http://androidxref.com/source/xref/frameworks/base/opengl/tests/gl2_yuvtex/gl2_yuvtex.cpp#193 
you can find a real example.

> 
> 
> .. but ok ... if that's the case: cool! That's going to have very good 
> performance!
> 
> 
> But I suspect(!) you'll end up with a copy anyway ... so:
> If there is a copy involved, it doesn't really matter where/who/how you copy 
> 
>
> as long as you copy junks (and not pixels) ... and glTexImage is your friend 
> here :)
> 
> .... it shouldn't be that big of a deal ... it's not helping the raster-
> backend-performance ... but if you want to mix with video I guess there is no 
> way around it.
> 
> 

I also suspect that a copy is done, but it seems that copy is (much) faster 
than glTexImage, as I said android is using only eglCreateImageKHR for camera
and video frames.

> 
> 
>>  IMHO it should not be a huge problem.
>>  This is the way I'm planning to draw the raster images using OpenGL,
>>  but where is Android's multimedia frame texture in this equations ?
>>  To be able to display that texture I'm planning to add a new surface
>>  type which will be linked to a TLW, this is the draft for the
>>  interface:
>> 
>>  class QSurfaceTexture
>>  {
>>     public:
>>     virtual void paintTexture() = 0; // here the widget will draw the
>>  texture by itself.
>>  }
>> 
>>  and I'm planning to add a few methods to QPlatformWindow, these
>>  methods will be used to register/unregister a texture surface and to
>>  iterate the textures.
>>  class QPlatformWindow
>>  {
>>  ....
>>    /// this function will be used to link a new surface texture to a TLW
>>    /// order is used to specify the drawing order,
>>    ///  - negative values (or 0) means the texture will be draw under the 
> TLW
>>  ///  - positive values means the texture will be draw over the TLW virtual
>>  void addSurfaceTexture(QSurfaceTexture *surface, int order = 0 ); virtual
>>  void removeSurfaceTexture(QSurfaceTexture *surface);
>> 
>>    /// gets the surfaces textures ordered by order value
>>    virtual const QList<QSurfaceTexture *> & 
> belowSurfaceTextures();
>>    virtual const QList<QSurfaceTexture *> & 
> aboveSurfaceTextures();
>>  }
>> 
>>  So, how a qt android multimedia widget will look like ?
>>  class QAndroidMediaSurfaceTextureWidget: public QWidget, public
>>  QSurfaceTexture, public AndroidSurfaceTexture
>>  {
>>  public:
>>        QAndroidMediaSurfaceTextureWidget( ... )
>>        {
>>           platformWindow()->addSurfaceTexture(this, -1); // the texture
>>  will be draw under the widget
>>        }
>> 
>>       void paintEvent ( QPaintEvent * event )
>>       {
>>         ...
>>          p.fill(qRgba(255,255,255,255)); // the widget must be filled
>>  with a transparent color, otherwise the texture behind it will not be
>>  visible.
>>       }
>> 
>>       void paintTexture()
>>       {
>>           glBindTexture(GL_TEXTURE_2D, androidTextureName);
>>           // draw the texture
>>       }
>>  }
>> 
>>  then a simple window manager will paint all TLW on the window surface:
>> 
>>  drawWindows()
>>  {
>>     eglMakeCurrent(...);
>>     foreach ( QPlatformWindow * window, m_windows)
>>     {
>>           foreach (QSurfaceTexture * surfaceTexture,
>>  window->belowSurfaceTextures())
>>                 surfaceTexture->paintTexture(); // first draw all below
>>  textures
>> 
>>          drawWindow(window); // update texture for that window, than draw 
> it.
>> 
>>           foreach (QSurfaceTexture * surfaceTexture,
>>  window->aboveSurfaceTextures())
>>                 surfaceTexture->paintTexture(); // last draw all below
>>  textures }
>>     eglSwapBuffers(...);
>>  }
>> 
>>  Now let's go back to the OpenGL rendering issue. Instead to use a
>>  windows surface, I'm thinking to switch to a pixelbuffer surface
>>  convert it to a texture (using glCopyTexImage2D ? ) then draw the
>>  texture to window surface as the other TLWs.
> 
> 
> No :) If you use OpenMAX AL you can let it render into a EGLSurface 
> 
> 
> And then there are functions to get a texture out of that,
> eglCreateImageKHR   <= this is one of the possible friends
> (this one would mean you first create a texture, map a EGLSurface around it, 
> and give that to openmax, there are other functions for the other way around)
> 
> 
> and then all you need to do is to render it using very limited opengl code.
> 
> There is already a QtMultimedia-backend using OpenMAX AL, it's for symbian 
> tough!... Not sure how much code-reuse is possible but it's definitely a 
> very 
> good reference!
> 
> qt-mobility/plugins/multimedia/symbian/openmaxal
> 
> 
> 
> The awesome part of this is that the only data going over the GPU-bus is the 
> encoded(!) file/data   (aka. zero-copy) 
> .... and the GPU-bus is one of the weak links on all platforms.
> 
> 
> 
> For the fist case: raster:
> If you go to way to copy/map the QImage's to EGL/GL resources, then 99% of 
> all 
> openmax al code should be reusable.
> 
> 

I think you misunderstand the last part which is not about OpenMAX.

 >>  Now let's go back to the OpenGL rendering issue. Instead to use a
>>  windows surface, I'm thinking to switch to a pixelbuffer surface
>>  convert it to a texture (using glCopyTexImage2D ? ) then draw the
>>  texture to window surface as the other TLWs.

Here I'm trying to say that if I'm going to use the only window EGLSurface, 
to paint the textures, then there is no *window* EGLSurface available for 
the QGLWidget, so for QGLWidget, I'm hopping to be able to use a *pixelbuffer*
EGLSurface, then convert it into a texture, then paint it to window EGLSurface.

Many thanks for your feedback !

> 
> 
>> 
>>  Any comments, suggestions and *help* will be very appreciated.
>> 
>>  Cheers,
>>  BogDan.
>> 
>> 
>>  [1] http://mail.kde.org/pipermail/necessitas-devel/2012-May/000900.html
>>  [2] http://code.google.com/p/android-lighthouse/issues/detail?id=4#c28
>>  [3] http://code.google.com/p/android-lighthouse/issues/detail?id=4#c23
>>  [4]
>>  http://groups.google.com/group/android-developers/browse_thread/thread/840e
>>  4e20faab5c31 ,
>>  https://developer.qualcomm.com/forum/qdevnet-forums/mobile-gaming-graphics-
>>  optimization-adreno/1969 _______________________________________________
>>  Necessitas-devel mailing list
>>  Necessitas-devel at kde.org
>>  https://mail.kde.org/mailman/listinfo/necessitas-devel
> _______________________________________________
> Necessitas-devel mailing list
> Necessitas-devel at kde.org
> https://mail.kde.org/mailman/listinfo/necessitas-devel
> > 