Qt Multimedia status on Android
BogDan
bog_dan_ro at yahoo.com
Thu May 3 19:12:57 UTC 2012
Hi Harri,
----- Original Message -----
> From: Harri Pasanen <harri at mpaja.com>
> To: necessitas-devel at kde.org
> Cc:
> Sent: Thursday, May 3, 2012 9:49 PM
> Subject: Re: Qt Multimedia status on Android
>
> Hats off for trying. ;)
>
> However, I think that for many uses a lower resolution would be enough.
> For symbian people are used to transcoding videos to 640x360. Also
> possibly there is a video format other than H264 that does not use
> YUV420P so decoding that would be faster? Did you look into that?
>
I didn't look into that direction, because H264 is a must for these days :)
And the people will wonder why qt is not able to play a 720p H264 video
as long as Android's player can play it with no problem.
>
> Btw. I was also looking into ffmpeg a while ago and noticed that there
> had been a split to libav, which ubuntu uses. I wonder what is the
> difference, is ffmpeg more optimized for ARM?
>
Yes, ffmpeg seems to be highly optimized for ARM, the decoding on ARMv7
with neon instructions is much faster than decoding on ARMv5 or ARMv7.
AFAIK libav has no stagefright support.
>
> Just curious,
>
> Harri
>
Thanks !
Cheers,
BogDan.
>
> On 05/03/2012 06:07 PM, BogDan Vatra wrote:
>> Hello everyone,
>>
>> For a couple of weeks me and a good friend tried to provide basic
>> multimedia for Qt on
>> Android, but sadly we didn't succeed at all, so I'd like to share
> with
>> you our findings.
>>
>> Our first track (and the most important one) was getting ffmpeg or
>> gstream compiled
>> on Android and use it to encode/decode the videos, I was misled by
> MoboPlayer[1]
>> which claimed to use ffmpeg to do the job, so everything seems to good
>> to be true.
>> After a long period of time of hard working, we manage to get ffmpeg
>> compiled on all android
>> supported processors (armv5, armv7 and armv7 with neon) [2], and to
>> create a simple video
>> player on top of ffmpeg, then we've discovered that arm processors are
>> not fast enough to
>> decode H264 frames, my HTC Desire HD phone (with a 1Ghz processor) was
>> not able to
>> decode a 720p frame in less than 90ms (using amrv7+neon), 150ms using
>> armv7 (without
>> neon) and 300ms using armv5 which is not enough to have a realtime
>> decoding.Even worse
>> after the decoding is finished the image still needs to be converted
>> from YUV420P to RGB,
>> this operations takes ages ! On my phone it takes from 300ms to 600ms
>> ! So, what we've
>> done wrong and why MoboPlayer is so damn fast? After I recover from
>> that shock I started to
>> investigate why MoboPlayer is that fast, soon I've discover that
>> MoboPlayer is cheating
>> big time! They are not using ffmpeg to decode the video ! They are
>> using Android's media
>> player for this job and they are using ffmpeg ONLY to get the
>> subtitles and to play other
>> formats which are not supported by Android. This discovery was a
>> disaster for me, because I
>> could not believe my eyes, I begin to search for another player which
>> is open source and
>> which I can check what is doing under the hood, I found Dolphin Player
>> [3][4], after I install it
>> my fears came true, the player was not able to play in realtime none
>> of my videos recorder
>> with my phone ! I dig deeper intro ffmpeg sources than I found that
>> they can use Android
>> stagefright[5] library to decode H264 frames, sadly this library can
>> not be used on all Android
>> devices and it only decodes to a YUV420P image, so we still need to
>> convert it to RGB.
>>
>> After I recovered from that heart attack I begin to investigate how
>> the hell Android is doing it that
>> fast, and if I can cheat OpenMAX to render into a buffer which I can
>> read it instead to render into
>> an Android surface. After I check almost all OpenMAX implementation I
>> found that it is possible
>> to send a SurfaceTextureClient[7]object to OpenMAX[6]. Again it seemed
>> to be too easy and too
>> good to be true, the same technique could be used also for camera
>> preview, even it needed non-public
>> APIs to be used in order to get it working (by the end of this year
>> all major Android vendors will have
>> upgrades to 4.x)!
>> Basically what I needed to do was just to implement ISurfaceTexture
>> interface[8] and to read the image.
>> The Image should contain one of these PixelFormat[9] image. Of course
>> nothing is that easy as it sounds,
>> and again it was a dead end, because Android is using *NONE* of these
>> pixel formats [10] !
>> For the name of God why somebody uses an unknown (useless) pixel
>> format ? So, I started to dig
>> deeper and deeper, and after a long and painful period of time, I
>> think I found the answer:
>> These images uses some "special" [11] pixel formats which are
> hardware
>> vendor specific, these images
>> can be pushed to the video card and the video card converts them very
>> very fast into a gl texture, practically
>> Android uses this textures to draw the frames onto a Surface, is
>> similar with this[12] test example. Now
>> the problem is how to use this texture in Qt? In order to display it
>> into a Qt 4.x application we need somehow
>> to get the pixels, we can use it to fill a FBO and read the pixels
>> using glReadPixels, but glReadPixels
>> is way too slow, or we can use it to fill a PBO which is a little bit
>> faster but not enough, actually is not
>> even close to what we need for a realtime video play !
>>
>> So whats next ? Sadly I have no idea how to create a realtime video
>> recoder/player on Android using
>> Qt 4.x ! So if nobody comes with another idea, I'm going to remove
>> multimedia plugin from alpha4 release.
>> It seems Qt 5.x will be able to use the texture directly [13] but I'm
>> not very sure !
>> Alpha4 is the last release were it comes with new features for Qt 4.x
>> after this release next releases
>> which will target Qt 4.x will contain only bugfixes. After we'll ship
>> alpha4 I'll focus on getting Qt 5 ported.
>>
>>
>> Cheers,
>> BogDan.
>>
>>
>> [1] https://play.google.com/store/apps/details?id=com.clov4r.android.nil
>> [2] https://gitorious.org/android-ffmpeg/android-ffmpeg
>> [3] http://code.google.com/p/dolphin-player/
>> [4] https://play.google.com/store/apps/details?id=com.broov.player
>> [5]
> https://gitorious.org/android-ffmpeg/android-ffmpeg/blobs/master/libavcodec/libstagefright.cpp
>> [6]
> http://androidxref.com/source/xref/system/media/wilhelm/src/android/MediaPlayer_to_android.cpp#android_Player_setNativeWindow
>> [7]
> http://androidxref.com/source/xref/frameworks/base/include/gui/SurfaceTextureClient.h
>> [8]
> http://androidxref.com/source/xref/frameworks/base/include/gui/ISurfaceTexture.h
>> [9]
> http://androidxref.com/source/xref/frameworks/base/include/ui/PixelFormat.h#63
>> [10]
> http://androidxref.com/source/xref/system/core/include/system/graphics.h#43
>> [11]
> http://androidxref.com/source/xref/system/core/include/system/graphics.h#54
>> [12]
> http://androidxref.com/source/xref/frameworks/base/opengl/tests/gl2_yuvtex/gl2_yuvtex.cpp
>> [13] http://qt-project.org/wiki/Qt-5-Alpha
>> _______________________________________________
>> Necessitas-devel mailing list
>> Necessitas-devel at kde.org
>> https://mail.kde.org/mailman/listinfo/necessitas-devel
>
> _______________________________________________
> Necessitas-devel mailing list
> Necessitas-devel at kde.org
> https://mail.kde.org/mailman/listinfo/necessitas-devel
>
More information about the Necessitas-devel
mailing list