Qt Multimedia status on Android

Thu May 3 18:57:16 UTC 2012

Hi,

Seems to use same trick as MoboPlayer :(, is very easy to check,
just try to record a small video using your phone, then play it
with the player, while the player is running run "adb shell top -m 5"
command and check the output, if "/system/bin/mediaserver" uses
more than 10% it means that the player is not using ffmpeg for
decoding.

Cheers,
BogDan.

2012/5/3 Scott Aron Bloom <Scott.Bloom at onshorecs.com>:
> Have you looked at rockplayer...
>
> It "definitely" uses ffmpeg, ok no real proof, except it can videos that
> the built in hardware based decoding cant, and it sucks down battery
> when in software mode.
>
> Scott
>
> -----Original Message-----
> From: android-qt at googlegroups.com [mailto:android-qt at googlegroups.com]
> On Behalf Of BogDan Vatra
> Sent: Thursday, May 03, 2012 9:07 AM
> To: android-qt; Necessitas
> Subject: Qt Multimedia status on Android
>
> Hello everyone,
>
>  For a couple of weeks me and a good friend tried to provide basic
> multimedia for Qt on Android, but sadly we didn't succeed at all, so I'd
> like to share with you our findings.
>
>  Our first track (and the most important one)  was getting ffmpeg or
> gstream compiled on Android and use it to encode/decode the videos, I
> was misled by MoboPlayer[1] which claimed to use ffmpeg to do the job,
> so everything seems to good to be true.
> After a long period of time of hard working, we manage to get ffmpeg
> compiled on all android supported processors (armv5, armv7 and armv7
> with neon) [2], and to create a simple video player on top of ffmpeg,
> then we've discovered that arm processors are not fast enough to decode
> H264 frames, my HTC Desire HD phone (with a 1Ghz processor) was not able
> to decode a 720p frame in less than 90ms (using amrv7+neon), 150ms using
> armv7 (without
> neon) and 300ms using armv5 which is not enough to have a realtime
> decoding.Even worse after the decoding is finished the image still needs
> to be converted from YUV420P to RGB, this operations takes ages ! On my
> phone it takes from 300ms to 600ms ! So, what we've done wrong and why
> MoboPlayer is so damn fast? After I recover from that shock I started to
> investigate why MoboPlayer is that fast, soon I've discover that
> MoboPlayer is cheating big time! They are not using ffmpeg to decode the
> video ! They are using Android's media player for this job and they are
> using ffmpeg ONLY to get the subtitles and to play other formats which
> are not supported by Android. This discovery was a disaster for me,
> because I could not believe my eyes, I begin to search for another
> player which is open source and which I can check what is doing under
> the hood, I found Dolphin Player [3][4], after I install it my fears
> came true, the player was not able to play in realtime none of my videos
> recorder with my phone ! I dig deeper intro ffmpeg sources than I found
> that they can use Android stagefright[5] library to decode H264 frames,
> sadly this library can not be used on all Android devices and it only
> decodes to a YUV420P image, so we still need to convert it to RGB.
>
> After I recovered from that heart attack I begin to investigate how the
> hell Android is doing it that fast, and if I can cheat OpenMAX to render
> into a buffer which I can read it instead to render into an Android
> surface. After I check almost all OpenMAX implementation I found that it
> is possible to send a SurfaceTextureClient[7]object to OpenMAX[6]. Again
> it seemed to be too easy and too good to be true, the same technique
> could be used also for camera preview, even it needed non-public APIs to
> be used in order to get it working (by the end of this year all major
> Android vendors will have upgrades to 4.x)!
> Basically what I needed to do was just to implement ISurfaceTexture
> interface[8] and to read the image.
> The Image should contain one of these PixelFormat[9] image. Of course
> nothing is that easy as it sounds, and again it was a dead end, because
> Android is using *NONE* of these pixel formats [10] !
> For the name of God why somebody uses an unknown (useless) pixel format
> ? So, I started to dig deeper and deeper, and after a long and painful
> period of time, I think I found the answer:
> These images uses some "special" [11] pixel formats which are hardware
> vendor specific, these images can be pushed to the video card and the
> video card converts them very very fast into a gl texture, practically
> Android uses this textures to draw the frames onto a Surface, is similar
> with this[12] test example. Now the problem is how to use this texture
> in Qt? In order to display it into a Qt 4.x application we need somehow
> to get the pixels, we can use it to fill a FBO and read the pixels using
> glReadPixels, but glReadPixels is way too slow, or we can use it to fill
> a PBO which is a little bit faster but not enough, actually is not even
> close to what we need for a realtime video play !
>
> So whats next ? Sadly I have no idea how to create a realtime video
> recoder/player on Android using Qt 4.x ! So if nobody comes with another
> idea, I'm going to remove multimedia plugin from alpha4 release.
> It seems Qt 5.x will be able to use the texture directly [13] but I'm
> not very sure !
> Alpha4 is the last release were it comes with new features for Qt 4.x
> after this release next releases which will target Qt 4.x will contain
> only bugfixes. After we'll ship
> alpha4 I'll focus on getting Qt 5 ported.
>
>
> Cheers,
> BogDan.
>
>
> [1] https://play.google.com/store/apps/details?id=com.clov4r.android.nil
> [2] https://gitorious.org/android-ffmpeg/android-ffmpeg
> [3] http://code.google.com/p/dolphin-player/
> [4] https://play.google.com/store/apps/details?id=com.broov.player
> [5]
> https://gitorious.org/android-ffmpeg/android-ffmpeg/blobs/master/libavco
> dec/libstagefright.cpp
> [6]
> http://androidxref.com/source/xref/system/media/wilhelm/src/android/Medi
> aPlayer_to_android.cpp#android_Player_setNativeWindow
> [7]
> http://androidxref.com/source/xref/frameworks/base/include/gui/SurfaceTe
> xtureClient.h
> [8]
> http://androidxref.com/source/xref/frameworks/base/include/gui/ISurfaceT
> exture.h
> [9]
> http://androidxref.com/source/xref/frameworks/base/include/ui/PixelForma
> t.h#63
> [10]
> http://androidxref.com/source/xref/system/core/include/system/graphics.h
> #43
> [11]
> http://androidxref.com/source/xref/system/core/include/system/graphics.h
> #54
> [12]
> http://androidxref.com/source/xref/frameworks/base/opengl/tests/gl2_yuvt
> ex/gl2_yuvtex.cpp
> [13] http://qt-project.org/wiki/Qt-5-Alpha