Puttin' on the GL(itz)
BogDan
bog_dan_ro at yahoo.com
Tue May 15 15:40:25 UTC 2012
----- Original Message -----
> From: Thomas Senyk <thomas.senyk at nokia.com>
> To: BogDan <bog_dan_ro at yahoo.com>; ext BogDan Vatra <taipanromania at gmail.com>
> Cc: "necessitas-devel at kde.org" <necessitas-devel at kde.org>
> Sent: Tuesday, May 15, 2012 5:47 PM
> Subject: Re: Puttin' on the GL(itz)
>
> On Tuesday, May 15, 2012 07:30:15 AM ext BogDan wrote:
>> Hi Thomas,
>>
>> > On Sunday, May 06, 2012 04:15:03 PM ext BogDan Vatra wrote:
>> >> Hello everyone,
>> >>
>> >> A few days ago I wrote an mail about "Qt Multimedia status
> on
>> >>
>> >> Android" [1], because I don't like to give up that easy,
> I tried
>> >
>> > to
>> >
>> >> find another solution and I think I found one (at least for
>> >> android-14+). Also some people post other solutions, and I'd
> like to
>> >> thank them very much! Their methods can be used for older android
>> >> versions.
>> >>
>> >> At the end of that email, I complained about the fact that
> there is
>> >>
>> >> no fast way to read the texture from graphics memory back to the
>> >> system memory.
>> >>
>> >> After several days of reflection (and more investigations), it
>> >>
>> >> striked me another idea: to move everything in video memory,
> instead
>> >> to read textures from the video memory.
>> >>
>> >> Before we go further, I'd like to make some clarifications :
>> >> * in order to understand this mail, first, you must read
> "Qt
>> >>
>> >> Multimedia status on Android" [1] !
>> >>
>> >> * this is just a draft, the is no code behind it, so, I
> don't know if
>> >>
>> >> it's going to work or not, just my thoughts!
>> >>
>> >> * my OpenGL silks are near zero, any help will be very
> appreciated,
>> >>
>> >> also, if anyone spots any mistakes *please* reply to this mail !
>> >>
>> >> Currently Qt on Android uses two methods to draw the controls:
> raster
>> >> or opengl rendering.
>> >> OpenGL rendering is done using your device graphics card, it
> doesn't
>> >> seem to be the best solution [2],[3] to draw controls, but is
> good for
>> >> complex animations. It also have a big problem, currently it is
>> >> limited to a single Top Level Widget (I'll use TLW
> abbreviation from
>> >> now on) because Android OpenGL is limited to a single window
> surface
>> >> per process. Let's forget about OpenGL for now, we'll
> come back to
>> >
>> > it
>> >
>> >> latter.
>> >> Raster rendering is done using a QImage for every TLW, is precise
> and
>> >> fast (if you don't have complex drawings and/or animations).
> So we
>> >> have a few QImage object which must be converted into gl
> textures,
>> >> then we can draw them to the windows surface, here I found that
> there
>> >> are two methods to do it, first one is to use glTex[Sub]Image*
> (this
>> >> one seems to be a little slow [4] on some devices, so maybe we
> should
>> >> avoid it ?) and another one is to use
>> >> eglCreateImageKHR/glEGLImageTargetTexture2DOES which seems much
>> >> faster, the problem with the second one is that we must use a few
>> >> non-public APIs to allocate the graphics buffer, even so, if is
> faster
>> >> than glTex[Sub]Image,
>> >
>> > Does this mean you can get a eglImage sitting on application-space
> memory
>> > (aka, QImage) without a copy?
>>
>> No, you can't use QImage, you need to create a special graphics buffer
>>
> http://androidxref.com/source/xref/frameworks/base/include/ui/GraphicBuffer.
>> h But you can create a QImage which shares the same memory.
>>
>> > Technically it's possible on a unified-memory system ... i guess
> ...,
>> > I'm just
>> > wondering because I've never seen any platform which can do that
>> > (Not even a platform which wasn't caring about memory management
> at all
>> > (e.g.
>> > vxworks))
>> >
>> >
>> > (fyi: I've used eglCreateImageKHR before ... it's main
> intention is to
>> > bind
>> > EGL and GL resources ... not software-resources
>> > ... but who knows ;)
>>
>> I don't know if the same memory is also used by the graphics card, but
> this
>> is the way android is doing with video and camera frames
>> Here
>> http://androidxref.com/source/xref/frameworks/base/opengl/tests/gl2_yuvtex/
>> gl2_yuvtex.cpp#193 you can find a real example.
>>
>> > .. but ok ... if that's the case: cool! That's going to have
> very good
>> > performance!
>> >
>> >
>> > But I suspect(!) you'll end up with a copy anyway ... so:
>> > If there is a copy involved, it doesn't really matter
> where/who/how you
>> > copy
>> >
>> >
>> > as long as you copy junks (and not pixels) ... and glTexImage is your
>> > friend here :)
>> >
>> > .... it shouldn't be that big of a deal ... it's not helping
> the raster-
>> > backend-performance ... but if you want to mix with video I guess
> there is
>> > no way around it.
>>
>> I also suspect that a copy is done, but it seems that copy is (much) faster
>> than glTexImage, as I said android is using only eglCreateImageKHR for
>> camera and video frames.
>>
>> >> IMHO it should not be a huge problem.
>> >> This is the way I'm planning to draw the raster images using
> OpenGL,
>> >> but where is Android's multimedia frame texture in this
> equations ?
>> >> To be able to display that texture I'm planning to add a new
> surface
>> >> type which will be linked to a TLW, this is the draft for the
>> >> interface:
>> >>
>> >> class QSurfaceTexture
>> >> {
>> >>
>> >> public:
>> >> virtual void paintTexture() = 0; // here the widget will draw
> the
>> >>
>> >> texture by itself.
>> >> }
>> >>
>> >> and I'm planning to add a few methods to QPlatformWindow,
> these
>> >> methods will be used to register/unregister a texture surface and
> to
>> >> iterate the textures.
>> >> class QPlatformWindow
>> >> {
>> >> ....
>> >>
>> >> /// this function will be used to link a new surface texture to
> a TLW
>> >> /// order is used to specify the drawing order,
>> >> /// - negative values (or 0) means the texture will be draw
> under the
>> >
>> > TLW
>> >
>> >> /// - positive values means the texture will be draw over the
> TLW
>> >> virtual
>> >> void addSurfaceTexture(QSurfaceTexture *surface, int order = 0 );
>> >> virtual
>> >> void removeSurfaceTexture(QSurfaceTexture *surface);
>> >>
>> >> /// gets the surfaces textures ordered by order value
>> >> virtual const QList<QSurfaceTexture *> &
>> >
>> > belowSurfaceTextures();
>> >
>> >> virtual const QList<QSurfaceTexture *> &
>> >
>> > aboveSurfaceTextures();
>> >
>> >> }
>> >>
>> >> So, how a qt android multimedia widget will look like ?
>> >> class QAndroidMediaSurfaceTextureWidget: public QWidget, public
>> >> QSurfaceTexture, public AndroidSurfaceTexture
>> >> {
>> >>
>> >> public:
>> >> QAndroidMediaSurfaceTextureWidget( ... )
>> >> {
>> >> platformWindow()->addSurfaceTexture(this, -1); // the
> texture
>> >>
>> >> will be draw under the widget
>> >>
>> >> }
>> >>
>> >> void paintEvent ( QPaintEvent * event )
>> >> {
>> >> ...
>> >> p.fill(qRgba(255,255,255,255)); // the widget must be
> filled
>> >>
>> >> with a transparent color, otherwise the texture behind it will
> not be
>> >> visible.
>> >>
>> >> }
>> >>
>> >> void paintTexture()
>> >> {
>> >> glBindTexture(GL_TEXTURE_2D, androidTextureName);
>> >> // draw the texture
>> >> }
>> >>
>> >> }
>> >>
>> >> then a simple window manager will paint all TLW on the window
> surface:
>> >>
>> >> drawWindows()
>> >> {
>> >>
>> >> eglMakeCurrent(...);
>> >> foreach ( QPlatformWindow * window, m_windows)
>> >> {
>> >> foreach (QSurfaceTexture * surfaceTexture,
>> >>
>> >> window->belowSurfaceTextures())
>> >>
>> >> surfaceTexture->paintTexture(); // first draw
> all below
>> >>
>> >> textures
>> >>
>> >> drawWindow(window); // update texture for that window,
> than draw
>> >
>> > it.
>> >
>> >> foreach (QSurfaceTexture * surfaceTexture,
>> >>
>> >> window->aboveSurfaceTextures())
>> >>
>> >> surfaceTexture->paintTexture(); // last draw
> all below
>> >>
>> >> textures }
>> >>
>> >> eglSwapBuffers(...);
>> >>
>> >> }
>> >>
>> >> Now let's go back to the OpenGL rendering issue. Instead to
> use a
>> >> windows surface, I'm thinking to switch to a pixelbuffer
> surface
>> >> convert it to a texture (using glCopyTexImage2D ? ) then draw the
>> >> texture to window surface as the other TLWs.
>> >
>> > No :) If you use OpenMAX AL you can let it render into a EGLSurface
>> >
>> >
>> > And then there are functions to get a texture out of that,
>> > eglCreateImageKHR <= this is one of the possible friends
>> > (this one would mean you first create a texture, map a EGLSurface
> around
>> > it, and give that to openmax, there are other functions for the other
> way
>> > around)
>> >
>> >
>> > and then all you need to do is to render it using very limited opengl
>> > code.
>> >
>> > There is already a QtMultimedia-backend using OpenMAX AL, it's for
> symbian
>> > tough!... Not sure how much code-reuse is possible but it's
> definitely a
>> > very
>> > good reference!
>> >
>> > qt-mobility/plugins/multimedia/symbian/openmaxal
>> >
>> >
>> >
>> > The awesome part of this is that the only data going over the GPU-bus
> is
>> > the encoded(!) file/data (aka. zero-copy)
>> > .... and the GPU-bus is one of the weak links on all platforms.
>> >
>> >
>> >
>> > For the fist case: raster:
>> > If you go to way to copy/map the QImage's to EGL/GL resources,
> then 99% of
>> > all
>> > openmax al code should be reusable.
>>
>> I think you misunderstand the last part which is not about OpenMAX.
>>
>> >> Now let's go back to the OpenGL rendering issue. Instead to
> use a
>>
>> >> windows surface, I'm thinking to switch to a pixelbuffer
> surface
>> >> convert it to a texture (using glCopyTexImage2D ? ) then draw the
>> >> texture to window surface as the other TLWs.
>>
>> Here I'm trying to say that if I'm going to use the only window
> EGLSurface,
>> to paint the textures, then there is no *window* EGLSurface available for
>> the QGLWidget, so for QGLWidget, I'm hopping to be able to use a
>> *pixelbuffer* EGLSurface, then convert it into a texture, then paint it to
>> window EGLSurface.
>
> Sorry! my bad! :)
> Replace all my "EGLSurface" with "EGLImage" :)
>
> EGLImage is a wrapper around "any" EGL/GL resource.
>
> So yes, you don't need a EGLSurface!
> ... all you need is a EGL/GL resource you can map a EGLImage around
> ... a texture is the simplest one. pixelbuffer or FBO are other examples.
>
>
Ok, but how do I bind a context to the current rendering thread and to the
EGLImage, I suppose I can not use eglMakeCurrent right ?
> And EGLImage is what OpenMAX AL deals with
> (actually I don't know that, but OpenMAX IL(!) does, so I suspect OpenMAX AL
>
> does as well)
>
>
> but again: any copy (within the gpu or not) should not be necessary!
>
>
>
>>
>>
>> Many thanks for your feedback !
>>
>> >> Any comments, suggestions and *help* will be very appreciated.
>> >>
>> >> Cheers,
>> >> BogDan.
>> >>
>> >>
>> >> [1]
> http://mail.kde.org/pipermail/necessitas-devel/2012-May/000900.html
>> >> [2]
> http://code.google.com/p/android-lighthouse/issues/detail?id=4#c28
>> >> [3]
> http://code.google.com/p/android-lighthouse/issues/detail?id=4#c23
>> >> [4]
>> >>
> http://groups.google.com/group/android-developers/browse_thread/thread/8
>> >> 40e
>> >> 4e20faab5c31 ,
>> >>
> https://developer.qualcomm.com/forum/qdevnet-forums/mobile-gaming-graphi
>> >> cs-
>> >> optimization-adreno/1969
> _______________________________________________
>> >> Necessitas-devel mailing list
>> >> Necessitas-devel at kde.org
>> >> https://mail.kde.org/mailman/listinfo/necessitas-devel
>> >
>> > _______________________________________________
>> > Necessitas-devel mailing list
>> > Necessitas-devel at kde.org
>> > https://mail.kde.org/mailman/listinfo/necessitas-devel
>
More information about the Necessitas-devel
mailing list