Puttin' on the GL(itz)

Tue May 15 14:47:55 UTC 2012

On Tuesday, May 15, 2012 07:30:15 AM ext BogDan wrote:
> Hi Thomas,
> 
> > On Sunday, May 06, 2012 04:15:03 PM ext BogDan Vatra wrote:
> >>  Hello everyone,
> >> 
> >>    A few days ago I wrote an mail about "Qt Multimedia status on
> >> 
> >>  Android" [1], because I don't like to give up that easy, I tried
> > 
> > to
> > 
> >>  find another solution and I think I found one (at least for
> >>  android-14+). Also some people post other solutions, and I'd like to
> >>  thank them very much! Their methods can be used for older android
> >>  versions.
> >> 
> >>    At the end of that email, I complained about the fact that there is
> >> 
> >>  no fast way to read the texture from graphics memory back to the
> >>  system memory.
> >> 
> >>    After several days of reflection (and more investigations), it
> >> 
> >>  striked me another idea: to move everything in video memory, instead
> >>  to read textures from the video memory.
> >> 
> >>  Before we go further, I'd like to make some clarifications :
> >>   * in order to understand this mail, first, you must read "Qt
> >> 
> >>  Multimedia status on Android" [1] !
> >> 
> >>   * this is just a draft, the is no code behind it, so, I don't know if
> >> 
> >>  it's going to work or not, just my thoughts!
> >> 
> >>   * my OpenGL silks are near zero, any help will be very appreciated,
> >> 
> >>  also, if anyone spots any mistakes *please* reply to this mail !
> >>  
> >>  Currently Qt on Android uses two methods to draw the controls: raster
> >>  or opengl rendering.
> >>  OpenGL rendering is done using your device graphics card, it doesn't
> >>  seem to be the best solution [2],[3] to draw controls, but is good for
> >>  complex animations. It also have a big problem, currently it is
> >>  limited to a single Top Level Widget (I'll use TLW abbreviation from
> >>  now on) because Android OpenGL is limited to a single window surface
> >>  per process. Let's forget about OpenGL for now, we'll come back to
> > 
> > it
> > 
> >>  latter.
> >>  Raster rendering is done using a QImage for every TLW, is precise and
> >>  fast (if you don't have complex drawings and/or animations). So we
> >>  have a few QImage object which must be converted into gl textures,
> >>  then we can draw them to the windows surface, here I found that there
> >>  are two methods to do it, first one is to use glTex[Sub]Image* (this
> >>  one seems to be a little slow [4] on some devices, so maybe we should
> >>  avoid it ?) and another one is to use
> >>  eglCreateImageKHR/glEGLImageTargetTexture2DOES which seems much
> >>  faster, the problem with the second one is that we must use a few
> >>  non-public APIs to allocate the graphics buffer, even so, if is faster
> >>  than glTex[Sub]Image,
> > 
> > Does this mean you can get a eglImage sitting on application-space memory
> > (aka, QImage) without a copy?
> 
> No, you can't use QImage, you need to create a special graphics buffer
> http://androidxref.com/source/xref/frameworks/base/include/ui/GraphicBuffer.
> h But you can create a QImage which shares the same memory.
> 
> > Technically it's possible on a unified-memory system ... i guess ...,
> > I'm just
> > wondering because I've never seen any platform which can do that
> > (Not even a platform which wasn't caring about memory management at all
> > (e.g.
> > vxworks))
> > 
> > 
> > (fyi: I've used eglCreateImageKHR before ... it's main intention is to
> > bind
> > EGL and GL resources ... not software-resources
> >    ... but who knows ;)
> 
> I don't know if the same memory is also used by the graphics card, but this
> is the way android is doing with video and camera frames
> Here
> http://androidxref.com/source/xref/frameworks/base/opengl/tests/gl2_yuvtex/
> gl2_yuvtex.cpp#193 you can find a real example.
> 
> > .. but ok ... if that's the case: cool! That's going to have very good
> > performance!
> > 
> > 
> > But I suspect(!) you'll end up with a copy anyway ... so:
> > If there is a copy involved, it doesn't really matter where/who/how you
> > copy
> > 
> > 
> > as long as you copy junks (and not pixels) ... and glTexImage is your
> > friend here :)
> > 
> > .... it shouldn't be that big of a deal ... it's not helping the raster-
> > backend-performance ... but if you want to mix with video I guess there is
> > no way around it.
> 
> I also suspect that a copy is done, but it seems that copy is (much) faster
> than glTexImage, as I said android is using only eglCreateImageKHR for
> camera and video frames.
> 
> >>  IMHO it should not be a huge problem.
> >>  This is the way I'm planning to draw the raster images using OpenGL,
> >>  but where is Android's multimedia frame texture in this equations ?
> >>  To be able to display that texture I'm planning to add a new surface
> >>  type which will be linked to a TLW, this is the draft for the
> >>  interface:
> >>  
> >>  class QSurfaceTexture
> >>  {
> >> 
> >>     public:
> >>     virtual void paintTexture() = 0; // here the widget will draw the
> >> 
> >>  texture by itself.
> >>  }
> >>  
> >>  and I'm planning to add a few methods to QPlatformWindow, these
> >>  methods will be used to register/unregister a texture surface and to
> >>  iterate the textures.
> >>  class QPlatformWindow
> >>  {
> >>  ....
> >> 
> >>    /// this function will be used to link a new surface texture to a TLW
> >>    /// order is used to specify the drawing order,
> >>    ///  - negative values (or 0) means the texture will be draw under the
> > 
> > TLW
> > 
> >>  ///  - positive values means the texture will be draw over the TLW
> >>  virtual
> >>  void addSurfaceTexture(QSurfaceTexture *surface, int order = 0 );
> >>  virtual
> >>  void removeSurfaceTexture(QSurfaceTexture *surface);
> >> 
> >>    /// gets the surfaces textures ordered by order value
> >>    virtual const QList<QSurfaceTexture *> &
> > 
> > belowSurfaceTextures();
> > 
> >>    virtual const QList<QSurfaceTexture *> &
> > 
> > aboveSurfaceTextures();
> > 
> >>  }
> >>  
> >>  So, how a qt android multimedia widget will look like ?
> >>  class QAndroidMediaSurfaceTextureWidget: public QWidget, public
> >>  QSurfaceTexture, public AndroidSurfaceTexture
> >>  {
> >> 
> >>  public:
> >>        QAndroidMediaSurfaceTextureWidget( ... )
> >>        {
> >>           platformWindow()->addSurfaceTexture(this, -1); // the texture
> >> 
> >>  will be draw under the widget
> >> 
> >>        }
> >> 
> >>       void paintEvent ( QPaintEvent * event )
> >>       {
> >>         ...
> >>          p.fill(qRgba(255,255,255,255)); // the widget must be filled
> >> 
> >>  with a transparent color, otherwise the texture behind it will not be
> >>  visible.
> >> 
> >>       }
> >> 
> >>       void paintTexture()
> >>       {
> >>           glBindTexture(GL_TEXTURE_2D, androidTextureName);
> >>           // draw the texture
> >>       }
> >> 
> >>  }
> >>  
> >>  then a simple window manager will paint all TLW on the window surface:
> >>  
> >>  drawWindows()
> >>  {
> >> 
> >>     eglMakeCurrent(...);
> >>     foreach ( QPlatformWindow * window, m_windows)
> >>     {
> >>           foreach (QSurfaceTexture * surfaceTexture,
> >> 
> >>  window->belowSurfaceTextures())
> >> 
> >>                 surfaceTexture->paintTexture(); // first draw all below
> >> 
> >>  textures
> >> 
> >>          drawWindow(window); // update texture for that window, than draw
> > 
> > it.
> > 
> >>           foreach (QSurfaceTexture * surfaceTexture,
> >> 
> >>  window->aboveSurfaceTextures())
> >> 
> >>                 surfaceTexture->paintTexture(); // last draw all below
> >> 
> >>  textures }
> >> 
> >>     eglSwapBuffers(...);
> >> 
> >>  }
> >>  
> >>  Now let's go back to the OpenGL rendering issue. Instead to use a
> >>  windows surface, I'm thinking to switch to a pixelbuffer surface
> >>  convert it to a texture (using glCopyTexImage2D ? ) then draw the
> >>  texture to window surface as the other TLWs.
> > 
> > No :) If you use OpenMAX AL you can let it render into a EGLSurface
> > 
> > 
> > And then there are functions to get a texture out of that,
> > eglCreateImageKHR   <= this is one of the possible friends
> > (this one would mean you first create a texture, map a EGLSurface around
> > it, and give that to openmax, there are other functions for the other way
> > around)
> > 
> > 
> > and then all you need to do is to render it using very limited opengl
> > code.
> > 
> > There is already a QtMultimedia-backend using OpenMAX AL, it's for symbian
> > tough!... Not sure how much code-reuse is possible but it's definitely a
> > very
> > good reference!
> > 
> > qt-mobility/plugins/multimedia/symbian/openmaxal
> > 
> > 
> > 
> > The awesome part of this is that the only data going over the GPU-bus is
> > the encoded(!) file/data   (aka. zero-copy)
> > .... and the GPU-bus is one of the weak links on all platforms.
> > 
> > 
> > 
> > For the fist case: raster:
> > If you go to way to copy/map the QImage's to EGL/GL resources, then 99% of
> > all
> > openmax al code should be reusable.
> 
> I think you misunderstand the last part which is not about OpenMAX.
> 
>  >>  Now let's go back to the OpenGL rendering issue. Instead to use a
> 
> >>  windows surface, I'm thinking to switch to a pixelbuffer surface
> >>  convert it to a texture (using glCopyTexImage2D ? ) then draw the
> >>  texture to window surface as the other TLWs.
> 
> Here I'm trying to say that if I'm going to use the only window EGLSurface,
> to paint the textures, then there is no *window* EGLSurface available for
> the QGLWidget, so for QGLWidget, I'm hopping to be able to use a
> *pixelbuffer* EGLSurface, then convert it into a texture, then paint it to
> window EGLSurface.

Sorry! my bad! :)
Replace all my "EGLSurface" with "EGLImage"  :)

EGLImage is a wrapper around "any" EGL/GL resource.

So yes, you don't need a EGLSurface! 
... all you need is a EGL/GL resource you can map a EGLImage around
     ... a texture is the simplest one. pixelbuffer or FBO are other examples.

And EGLImage is what OpenMAX AL deals with
(actually I don't know that, but OpenMAX IL(!) does, so I suspect OpenMAX AL 
does as well)

but again: any copy (within the gpu or not)  should not be necessary!

> 
> 
> Many thanks for your feedback !
> 
> >>  Any comments, suggestions and *help* will be very appreciated.
> >>  
> >>  Cheers,
> >>  BogDan.
> >>  
> >>  
> >>  [1] http://mail.kde.org/pipermail/necessitas-devel/2012-May/000900.html
> >>  [2] http://code.google.com/p/android-lighthouse/issues/detail?id=4#c28
> >>  [3] http://code.google.com/p/android-lighthouse/issues/detail?id=4#c23
> >>  [4]
> >>  http://groups.google.com/group/android-developers/browse_thread/thread/8
> >>  40e
> >>  4e20faab5c31 ,
> >>  https://developer.qualcomm.com/forum/qdevnet-forums/mobile-gaming-graphi
> >>  cs-
> >>  optimization-adreno/1969 _______________________________________________
> >>  Necessitas-devel mailing list
> >>  Necessitas-devel at kde.org
> >>  https://mail.kde.org/mailman/listinfo/necessitas-devel
> > 
> > _______________________________________________
> > Necessitas-devel mailing list
> > Necessitas-devel at kde.org
> > https://mail.kde.org/mailman/listinfo/necessitas-devel