Puttin' on the GL(itz)

[BogDan Vatra]

Hello everyone,

  A few days ago I wrote an mail about "Qt Multimedia status on
Android" [1], because I don't like to give up that easy, I tried to
find another solution and I think I found one (at least for
android-14+). Also some people post other solutions, and I'd like to
thank them very much! Their methods can be used for older android
versions.
  At the end of that email, I complained about the fact that there is
no fast way to read the texture from graphics memory back to the
system memory.
  After several days of reflection (and more investigations), it
striked me another idea: to move everything in video memory, instead
to read textures from the video memory.

Before we go further, I'd like to make some clarifications :
 * in order to understand this mail, first, you must read "Qt
Multimedia status on Android" [1] !
 * this is just a draft, the is no code behind it, so, I don't know if
it's going to work or not, just my thoughts!
 * my OpenGL silks are near zero, any help will be very appreciated,
also, if anyone spots any mistakes *please* reply to this mail !

Currently Qt on Android uses two methods to draw the controls: raster
or opengl rendering.
OpenGL rendering is done using your device graphics card, it doesn't
seem to be the best solution [2],[3] to draw controls, but is good for
complex animations. It also have a big problem, currently it is
limited to a single Top Level Widget (I'll use TLW abbreviation from
now on) because Android OpenGL is limited to a single window surface
per process. Let's forget about OpenGL for now, we'll come back to it
latter.
Raster rendering is done using a QImage for every TLW, is precise and
fast (if you don't have complex drawings and/or animations). So we
have a few QImage object which must be converted into gl textures,
then we can draw them to the windows surface, here I found that there
are two methods to do it, first one is to use glTex[Sub]Image* (this
one seems to be a little slow [4] on some devices, so maybe we should
avoid it ?) and another one is to use
eglCreateImageKHR/glEGLImageTargetTexture2DOES which seems much
faster, the problem with the second one is that we must use a few
non-public APIs to allocate the graphics buffer, even so, if is faster
than glTex[Sub]Image, IMHO it should not be a huge problem.
This is the way I'm planning to draw the raster images using OpenGL,
but where is Android's multimedia frame texture in this equations ?
To be able to display that texture I'm planning to add a new surface
type which will be linked to a TLW, this is the draft for the
interface:

class QSurfaceTexture
{
   public:
   virtual void paintTexture() = 0; // here the widget will draw the
texture by itself.
}

and I'm planning to add a few methods to QPlatformWindow, these
methods will be used to register/unregister a texture surface and to
iterate the textures.
class QPlatformWindow
{
....
  /// this function will be used to link a new surface texture to a TLW
  /// order is used to specify the drawing order,
  ///  - negative values (or 0) means the texture will be draw under the TLW
  ///  - positive values means the texture will be draw over the TLW
  virtual void addSurfaceTexture(QSurfaceTexture *surface, int order = 0 );
  virtual void removeSurfaceTexture(QSurfaceTexture *surface);

  /// gets the surfaces textures ordered by order value
  virtual const QList<QSurfaceTexture *> & belowSurfaceTextures();
  virtual const QList<QSurfaceTexture *> & aboveSurfaceTextures();
}

So, how a qt android multimedia widget will look like ?
class QAndroidMediaSurfaceTextureWidget: public QWidget, public
QSurfaceTexture, public AndroidSurfaceTexture
{
public:
      QAndroidMediaSurfaceTextureWidget( ... )
      {
         platformWindow()->addSurfaceTexture(this, -1); // the texture
will be draw under the widget
      }

     void paintEvent ( QPaintEvent * event )
     {
       ...
        p.fill(qRgba(255,255,255,255)); // the widget must be filled
with a transparent color, otherwise the texture behind it will not be
visible.
     }

     void paintTexture()
     {
         glBindTexture​(GL_TEXTURE_2D, androidTextureName);
         // draw the texture
     }
}

then a simple window manager will paint all TLW on the window surface:

drawWindows()
{
   eglMakeCurrent(...);
   foreach ( QPlatformWindow * window, m_windows)
   {
         foreach (QSurfaceTexture * surfaceTexture,
window->belowSurfaceTextures())
               surfaceTexture->paintTexture(); // first draw all below textures

        drawWindow(window); // update texture for that window, than draw it.

         foreach (QSurfaceTexture * surfaceTexture,
window->aboveSurfaceTextures())
               surfaceTexture->paintTexture(); // last draw all below textures
   }
   eglSwapBuffers(...);
}

Now let's go back to the OpenGL rendering issue. Instead to use a
windows surface, I'm thinking to switch to a pixelbuffer surface
convert it to a texture (using glCopyTexImage2D ? ) then draw the
texture to window surface as the other TLWs.

Any comments, suggestions and *help* will be very appreciated.

Cheers,
BogDan.


[1] http://mail.kde.org/pipermail/necessitas-devel/2012-May/000900.html
[2] http://code.google.com/p/android-lighthouse/issues/detail?id=4#c28
[3] http://code.google.com/p/android-lighthouse/issues/detail?id=4#c23
[4] http://groups.google.com/group/android-developers/browse_thread/thread/840e4e20faab5c31
, https://developer.qualcomm.com/forum/qdevnet-forums/mobile-gaming-graphics-optimization-adreno/1969