Qt Multimedia status on Android

Wed May 9 11:46:53 UTC 2012

On Thu, May 3, 2012 at 6:07 PM, BogDan Vatra <taipanromania at gmail.com> wrote:
> Hello everyone,
>
>  For a couple of weeks me and a good friend tried to provide basic
> multimedia for Qt on
> Android, but sadly we didn't succeed at all, so I'd like to share with
> you our findings.
>
>  Our first track (and the most important one)  was getting ffmpeg or
> gstream compiled
> on Android and use it to encode/decode the videos, I was misled by MoboPlayer[1]
> which claimed to use ffmpeg to do the job, so everything seems to good
> to be true.
> After a long period of time of hard working, we manage to get ffmpeg
> compiled on all android
> supported processors (armv5, armv7 and armv7 with neon) [2], and to
> create a simple video
> player on top of ffmpeg, then we've discovered that arm processors are
> not fast enough to
> decode H264 frames, my HTC Desire HD phone (with a 1Ghz processor) was
> not able to
> decode a 720p frame in less than 90ms (using amrv7+neon), 150ms using
> armv7 (without
> neon) and 300ms using armv5 which is not enough to have a realtime
> decoding.Even worse
> after the decoding is finished the image still needs to be converted
> from YUV420P to RGB,
> this operations takes ages ! On my phone it takes from 300ms to 600ms
> ! So, what we've
> done wrong and why MoboPlayer is so damn fast? After I recover from
> that shock I started to
> investigate why MoboPlayer is that fast, soon I've discover that
> MoboPlayer is cheating
> big time! They are not using ffmpeg to decode the video ! They are
> using Android's media
> player for this job and they are using ffmpeg ONLY to get the
> subtitles and to play other
> formats which are not supported by Android. This discovery was a
> disaster for me, because I
> could not believe my eyes, I begin to search for another player which
> is open source and
> which I can check what is doing under the hood, I found Dolphin Player
> [3][4], after I install it
> my fears came true, the player was not able to play in realtime none
> of my videos recorder
> with my phone ! I dig deeper intro ffmpeg sources than I found that
> they can use Android
> stagefright[5] library to decode H264 frames, sadly this library can
> not be used on all Android
> devices and it only decodes to a YUV420P image, so we still need to
> convert it to RGB.
>
> After I recovered from that heart attack I begin to investigate how
> the hell Android is doing it that
> fast, and if I can cheat OpenMAX to render into a buffer which I can
> read it instead to render into
> an Android surface. After I check almost all OpenMAX implementation I
> found that it is possible
> to send a SurfaceTextureClient[7]object to OpenMAX[6]. Again it seemed
> to be too easy and too
> good to be true, the same technique could be used also for camera
> preview, even it needed non-public
> APIs to be used in order to get it working (by the end of this year
> all major Android vendors will have
> upgrades to 4.x)!
> Basically what I needed to do was just to implement ISurfaceTexture
> interface[8] and to read the image.
> The Image should contain one of these PixelFormat[9] image. Of course
> nothing is that easy as it sounds,
> and again it was a dead end, because Android is using *NONE* of these
> pixel formats [10] !
> For the name of God why somebody uses an unknown (useless) pixel
> format ? So, I started to dig
> deeper and deeper, and after a long and painful period of time, I
> think I found the answer:
> These images uses some "special" [11] pixel formats which are hardware
> vendor specific, these images
> can be pushed to the video card and the video card converts them very
> very fast into a gl texture, practically
> Android uses this textures to draw the frames onto a Surface, is
> similar with this[12] test example. Now
> the problem is how to use this texture in Qt? In order to display it
> into a Qt 4.x application we need somehow
> to get the pixels, we can use it to fill a FBO and read the pixels
> using glReadPixels, but glReadPixels
> is way too slow, or we can use it to fill a PBO which is a little bit
> faster but not enough, actually is not
> even close to what we need for a realtime video play !
>
> So whats next ? Sadly I have no idea how to create a realtime video
> recoder/player on Android using
> Qt 4.x ! So if nobody comes with another idea, I'm going to remove
> multimedia plugin from alpha4 release.
> It seems Qt 5.x will be able to use the texture directly [13] but I'm
> not very sure !
> Alpha4 is the last release were it comes with new features for Qt 4.x
> after this release next releases
> which will target Qt 4.x will contain only bugfixes. After we'll ship
> alpha4 I'll focus on getting Qt 5 ported.
>
>
> Cheers,
> BogDan.
>
>
> [1] https://play.google.com/store/apps/details?id=com.clov4r.android.nil
> [2] https://gitorious.org/android-ffmpeg/android-ffmpeg
> [3] http://code.google.com/p/dolphin-player/
> [4] https://play.google.com/store/apps/details?id=com.broov.player
> [5] https://gitorious.org/android-ffmpeg/android-ffmpeg/blobs/master/libavcodec/libstagefright.cpp
> [6] http://androidxref.com/source/xref/system/media/wilhelm/src/android/MediaPlayer_to_android.cpp#android_Player_setNativeWindow
> [7] http://androidxref.com/source/xref/frameworks/base/include/gui/SurfaceTextureClient.h
> [8] http://androidxref.com/source/xref/frameworks/base/include/gui/ISurfaceTexture.h
> [9] http://androidxref.com/source/xref/frameworks/base/include/ui/PixelFormat.h#63
> [10] http://androidxref.com/source/xref/system/core/include/system/graphics.h#43
> [11] http://androidxref.com/source/xref/system/core/include/system/graphics.h#54
> [12] http://androidxref.com/source/xref/frameworks/base/opengl/tests/gl2_yuvtex/gl2_yuvtex.cpp
> [13] http://qt-project.org/wiki/Qt-5-Alpha
> _______________________________________________
> Necessitas-devel mailing list
> Necessitas-devel at kde.org
> https://mail.kde.org/mailman/listinfo/necessitas-devel

maybe it would be possible to use VLC? I've been told they have quite
a stable library.

Also they have a phonon backend, so that would help with the
applications that use it.

Aleix