Accelerating in my pocket

I started poking at the SMedia Glamo chip in the GTA02 this week. First I played with the Linux framebuffer driver and later with decoding MPEG in hardware, and now I have some code ready. I was challenged by messages like this on the Openmoko lists. Contrary to the opinion spreading accross these messages, we’re not doomed and we still have a graphics accelerator in a phone (which is coolness on its own). And it’s a quite hackable one.

I first had a look at libglamo code – a small library written some time ago by Chia-I Wu (olv) and Harald Welte (laf0rge) for accessing some of the Glamo’s submodules (engines). I asked the authors if I could use their code and release it under GPL and they liked the idea, so I stitched together libglamo and mplayer and added the necessary glue drivers. This wasn’t all straight forward because mplayer isn’t really prepared for doing decoding in hardware, even though some support was present. Today I uploaded my mplayer git tree here – see below what it can and cannot do. There’s lots more that can be improved but the basic stuff is there and seems to work. To clone, do this:

cg-clone git://repo.or.cz/mplayer/glamo.git

The Glamo fact sheet claims it can do MPEG-4 and H-263 encoding/decoding at 352×288, 30fps max and 640×480 at 12fps max. Since it also does all the scaling/rotation in hardware, I hoped I would be able to play a 352×288 video scaled to 640×480 at full frame-rate but this doesn’t seem to be the case. The decoding is pretty fast but the scaling takes a while and rotation adds another bit of overhead. That said, even if mplayer is not keeping up with the video’s frame-rate it still shows 0.0% CPU usage in top. There are still many obvious optimisations that can be done (and some less obvious that I don’t know about not being much into graphics). Usage considerations:

  • Pass “-vo glamo” to use the glamo driver. The driver should probably be a VIDIX subdriver in mplayer’s source but that would take much more work as VIDIX is very incomplete now, so glamo is a separate output driver (in particular vidix seems to support only “BES” (backend scaler?) type of hw acceleration, which the Glamo also does, but it does much more too). Like vidix, it requires root access to run (we should move the driver to the kernel once there exists a kernel API for video decoders – or maybe to X).
  • It only supports MPEG-4 videos, so you should recode if you want to watch something on the phone without using much CPU. H-263 would probably only require some trivial changes in the code. For completeness – MPEG-4 is not backwards compatible with MPEG1 or 2, it’s a separate codec. It’s the one used by most digital cameras and it can be converted to/from with Fabrice Bellard’s ffmpeg. A deblocking filter is supported by the Glamo but the driver doesn’t yet support it. For other codecs, “-vo glamo” will try to help in converting the decoded frames from YUV to RGB (untested), which is normally the last step of decoding.
  • The “glamo” driver can take various parameters. Add “:rotate=90″ to rotate (or 180 or 270) – the MPEG engine doesn’t know about the xrandr rotation and they won’t work together. Add “:nosleep” to avoid sleeping in mplayer – this yields slightly better FPS but takes up all your CPU, spinning.
  • Supports the “xover” output driver, pass “-vo xover:glamo” to use that (not very useful with a window manager that makes all windows full-screen anyway).
  • Only works with the 2.6.22.5 Openmoko kernels. There were some changes in openmoko 2.6.24 patches that disabled access to the MPEG engine but since we don’t have a bisectable git tree I can’t be bothered. UPDATE: A 2.6.24 patch here – note that it can eat your files, no responsibility assumed. I guess it can also be accounted for in mplayer, will check. My rant about lack of changes history in git is still valid – while I loved the switch to git, the SVN was being maintained better in this regard.
  • In the mplayer git tree linked above I enabled anonymous unmoderated push access so improvements are welcome and easy to get in.

With respect to the linux framebuffer poking, I wanted to see how much of the text console rendering can be moved to the hardware side and it seems the hw is not lacking anything (scrolling, filling rectagles, cursor) compared to the other accelerated video cards, and even the code already exists in Dodji Seketeli’s Xglamo. I’m sure sooner or later we’ll have it implemented in the kernel too. For now I got the framebuffer to use hardware cursor drawing (alas still with issues).

About these ads

40 Responses to “Accelerating in my pocket”

  1. Fredrik Wendt Says:

    Just wanted to say Thank you and awesome! I’m so greatful for your contributions to my free mobile life (soon to be anyway).

  2. Yorick Says:

    I’m not much of a programmer, but this seems GREAT!
    Does this mean we will be able to watch fullscreen movies on our FreeRunner?
    Even if nog, i’m very grateful you’ve taken the time to work on this!

  3. Mickey Says:

    Excellent, keep it up!

  4. IAnjo Says:

    Thanks alot! I was beginning to accept that I simply wouldn’t have decent video on my OM and you come along and change that :)

  5. doc Says:

    nice work! If the rotation is slowing things down could it be an option/selection in mplayer? It make it possible to allow people to have a higher frame rate at the loss of displayable.

    Cheers

  6. mutrox Says:

    Totally awesome, count with a tester when the freerunner arrives

  7. mutrox Says:

    BTW this picture is pure geekytechie porn, a naked freeruner playing video at fullscreen with the fps in the rear back in a terminal… simply beautyful

  8. balrog Says:

    Yorick, IAnjo: well, by all means it means you’ll be able to watch video without stressing the CPU. I’m not sure about decency because I couldn’t squeeze 20fps out of it at 640×480. There are some possibilities of doing more things concurrently, or overclocking the Glamo – but to have a true, reliable fullscreeen playback at full frame-rate, the only option is probably still the QVGA mode (320×240). I think the difference between QVGA and VGA is hardly noticeable at this screen size though.

    doc: yes, it’s completely optional, by default the rotation is disabled. And, frankly, if you have to recode the videos to a different codec anyway, you can do the rotation and scalling during recoding, on your PC.

    I hear that PDA users are already accustomised to recoding their videos, but admittedly the Nokia N800 wins because it does 800×480 at 20fps and even supports MPEG1/2.

    mutrox: my freerunner comes from before the new case was designed so it looks like a Neo1973 – that’s why it’s naked :p

  9. doc Says:

    Thanks for the response. again, nice work :)

  10. Mikko Rauhala Says:

    Ooo, what with all the Raster’s gloom about nobody having time to do mpeg4 decoding support and video playback being not on the maps, this is excellent news. I’m quite happy with kludged playback, just as long as there’s playback.

    Is your pondering about QVGA mode conjecture or have you tried it yet? It’d be just excellent if one could play (at least some) preexisting full framerate mpeg-4 files rotated and fullscreened if you just switch to QVGA first. (I presume QVGA mode on the LCD is still possible to do on the GTA02; pretty please?)

    Anyway, yeah, thanks, I worship you. :)

  11. i_m_bison Says:

    whats the video on mtv? :P

    good job! i wish i cud do something like that. what does it take to do all this? :D

  12. tuXXX Says:

    Good work on this! I think that the hardware is more powerful than many people think.

    I own a GP2X, and it’s able to decode xvid videos at nearly VGA quality with a good speed (~20 fps at default speed of 200MHz, fullspeed at 250).
    And this is only with two 200 (or 250 O/C) CPUs.

    I’m wondering if it’s possible to use both the glamo and the CPU, for example using the glamo chip for mpeg4 decoding and the CPU for scaling/rotation (rotating a VGA image with a 400MHz CPU should be ok, isn’t it?).

  13. Mikko Rauhala Says:

    tuXXX: No, as the major bottleneck is the CPU-GPU bus, you cannot bounce the data back-and-forth and expect it to perform. The QVGA mode, if possible, is the primary contender here for a smooth solution.

  14. balrog Says:

    Mikko: I hope we’ll eventually see more than just MPEG support on the Glamo (3D stuff etc). Regarding QVGA at this moment it’s a conjecture…
    and at this moment it’s confirmed. I’ve just tried QVGA and while the colours became a bit undersaturated and gamma seems to be too high, I got full 25fps playback both straight and with rotation, hw-scalled from 352×288 to 240×320 or 320×240. Unfortunately tslib seems to not have noticed the change and still reports touch coords in 0…640 range.

    i_m_bison: Darude_-_Feel_the_beat.mp4 (the video is ok, the music not so much:))

    tuXXX: I was wondering about that too, but rather MPEG decoding on the CPU and scaling/rotation on Glamo – this would allow us to do more stuff concurrently. The other way, as Mikko notes you transfer the video to the VRAM, then back to the RAM and then back to the VRAM and the bus becomes a bottleneck. In the current setup and normal usage I don’t agree with Raster that 7M/s is a bottleneck, it’s rather hard to saturate with normal X usage even coupled with SD card traffic etc.

  15. Mikko Rauhala Says:

    Thanks for the QVGA news :) Can the chip do programmable gamma and stuff to correct it a bit, or is there some other tweaking possible (ie. are some LCD initialization parameters for the QVGA mode to blame or something)? (Not that I have overly high quality requirements for this kind of use anyway; just hearing it basically works is quite dandy.)

    As for decoding on the CPU, I did find part of Raster’s doom and gloom credible in that the bus becomes a real bottleneck for just upping decompressed video frames to the GPU, given that the data transfer apparently ties the CPU up (and also conversely, using 100% of that bus bandwidth ties the CPU up 100%, no time for decompression). But for me it’s all hearsay anyway, feel free to provide experimental refutation ;P

  16. zub Says:

    Thanks for the work. It’s wonderful news. :D Makes me even more impatient to get my hands on FreeRunner.

  17. Jason Says:

    I am particularly interested in the support of Theora (http://www.theora.org/) in an attempt to avoid patent issues.

    How easy/hard is adding support for an additional codec?

  18. Fradeve Says:

    Simply I would to thank you for your work, really great! :D I add my interest for Theora codecs ad reported up by Jason…

  19. EdorFaus Says:

    Jason, Fradeve: With the assumption that the Glamo does not already support Theora, I believe it is a very safe bet that adding Theora support to it is (practically) impossible, seeing as it is a completely different codec.

    Note that this is when we are talking about the hardware decompression.

    Using the Glamo for scaling and rotation while decompressing Theora on the CPU is a different matter, and should work in much the same way as for MPEG-4, which was suggested for concurrency reasons. As mentioned, however, this might run into bandwidth problems on the FR.

    Additionally, I don’t know how CPU-intensive Theora is compared with MPEG-4, so this solution *might* be infeasible due to that.

  20. kriss Says:

    You rock! :-)

  21. Witek Says:

    what about writing real X driver with Xv support? It will be grate.

  22. balrog Says:

    Jason: EdorFaus is right I’m afraid – supporting any codec other than mpeg4 / H.263+ is only as easy as changing the hardware.

    Witek: Xglamo is the real X for the glamo with Xv support. Xv is implemented in hardware in Xglamo, but as far as I know Xv in general doesn’t support any particular codec (yet?), only stuff like scalling and YUV->RGB. If you use mplayer with -vo xv, it should just work.

  23. xeros Says:

    balrog: do you plan to merge yours git branch into official mplayer repo in near future?
    I’ve bought FR (but doesn’t have it yet) and I can’t test it yet but I think it shouldn’t make problems to the official branch as it’s another video output plugin and then it could be packaged in the OpenMoko package repos.

  24. kd8ikt Says:

    anyone have an ipkg?

  25. Treviño Says:

    I’m trying to compile this with the OM toolchain but I cant… Have you a working binary somewhere?

  26. balrog Says:

    xerox: I would like to but I’m unsure about libglamo. It’s required by the driver, but maybe it should be distributed separately from mplayer as it’s quite self-contained. Also lacking time at the moment.

    Treviño: I uploaded two binaries at ftp://balrog.myftp.org/mplayer/ built with different toolchains and with different options. One is the one I used and the other one was built by Sören Apel recently. Strangely, they behave differently on my GTA02A3, and Sören gets yet different behavior on his A5. To use Sören’s binary on my Neo I had to override the demuxer to use the QuickTime mp4 demuxer through commandline option. On the other hand Sören reports that it works out of the box on his Neo but there are some glitches and it uses more CPU (I think that’s related to the glamo interrupt hardware bug on A3, which keeps interrupts from using up CPU, need to investigate it some time).

  27. Treviño Says:

    Thanks, they work and I’ll be also able to compile a copy using the OM toolchain and your configuration. I’ve put it in the repo (mob branch).

  28. Treviño Says:

    Ehm, sorry I didn’t see that there was a config_cmd there already… I can’t understand why I’ve not seen it before, and I did looked for something like that! Maybe it was too late :|.

    Btw I’ve fixed a small Makefile issue :P.

    Then, I’ve recompiled mplayer with faad support and now I can also play low quality mp4 youtube videos without re-encoding them (so also in streaming). This is quite good!

    The only problem I’ve it’s that I can’t make them fullscreen so well; using the xover option I get that (also if often the videos are not using real proportions – any tip?) but they have unshaped pixels… Is there a way to zoom the video removing this noise as the Xv zoom does?

  29. Treviño Says:

    Since the ftp seems to be down, I’ve put a mplayer that I’ve compiled (it uses the same config by barlog, but it has the AAC decoding enabled) in a web space I’ve got; I’ve put there also a gta02 kernel patched with some andy glamo patches and with the one linked in this article (it works well, except suspend):
    mplayer-glamo.tar.bz2
    uImage-gta02-stable-glamo-video-patch%2bgitd744c88c149269b95ec068c8615e492375415d6d.bin

    With these files I’ve been able to play mp4 videos. I can also play (streaming) youtube mp4 videos, but only in low quality (i.e.: 176×144 @ 6fps – see link in the previous comment and try ID POiNqP4savI :P)
    Youtube would provide natively also mp4 at 480×360 @ 30fps. They obiouvsly have too many pixels per second for our chip, so if you play them you don’t see the video at all (just a striped view with the colors of the video) btw I was figuring that *maybe* mplayer could take from the file only some frames each second (10?) to make it viewable (I was thinking of sending to the glamo chip only a subset of all the frames, shouldn’t this allow playing?). However the -fps option doesn’t seem to do the work. Which is the problem?

  30. Neo FreeRunner - sprzęt i wydajność at harnir’s blog Says:

    [...] Brak FPU niestety daje się we znaki – FreeRunner nie przyda się tam, gdzie trzeba przeprowadzić dużo operacji na liczbach zmiennoprzecinkowych, jak łamanie szyfrów lub dużo grafiki 3D – chociaż, gdyby wykorzystać jakoś GPU karty graficznej… Cóż, zobaczymy w przyszłości, kiedy uda się uzyskać do niego pełen dostęp. [...]

  31. sledge Says:

    I am compiling the git sources using the Openmoko’s latest toolchain (from http://wiki.openmoko.org/wiki/Toolchain), but running # om-conf glamo
    gives:
    Source directory: /media/rootas/home/sledge/glamo
    Extra configure arguments:
    NOTE: Running /media/rootas/home/sledge/glamo/configure –build=i686-linux –host=arm-angstrom-linux-gnueabi –target=arm-angstrom-linux-gnueabi –prefix=/usr –exec_prefix=/usr –bindir=/usr/bin –sbindir=/usr/sbin –libexecdir=/usr/libexec –datadir=/usr/share –sysconfdir=/etc –sharedstatedir=/usr/com –localstatedir=/var –libdir=/usr/lib –includedir=/usr/include –oldincludedir=/usr/include –infodir=/usr/share/info –mandir=/usr/share/man –enable-mainainer-mode …
    Unknown parameter: –build=i686-linux
    FATAL: oe_runconf failed

  32. sledge Says:

    PS: my compile system is gentoo in x86 (Intel Santa Rosa i965 ICH8)

  33. Ed Says:

    Hi Treviño,

    I tried your kernel and the mplayer binary, but all i get is an scrambeled video output.
    audio is ok.
    The video look like it is out of sync (vertically)
    i am using a freerunner with:
    Found SMedia Glamo 3362, with 3365 core, rev A2

    Any idea on how to get it working?

    Kind regards,
    Ed

  34. IT Says:

    nice stuff

  35. Treviño Says:

    Ed, what video are you playing? If the video is too big (from the resolution+FPS point of view), you’ll get that behavior…

  36. Denis Says:

    How do I pass options such as rotate=90 to xover:glamo? Reading mplayer man page gave nothing…

  37. Andy Says:

    Great blog! Thank you for posting about the Openmoko platform. I think it is so exciting! Please check out my blog @ http://www.okmko.com and let me know what you think. Cheers!

  38. Radek Says:

    Hi, have found out solution to wrong colors in qvga. Hope it helps someone:

    * Install fbset program

    * Create /etc/fb.modes

    mode “vga”
    geometry 480 640 480 1280 16
    timings 40816 8 16 2 16 8 2
    rgba 5/11,6/5,5/0,0/0
    endmode

    mode “qvga”
    geometry 480 640 480 1280 16
    timings 100000 8 16 2 16 8 2
    rgba 5/11,6/5,5/0,0/0
    endmode

    Calling “fbset qvga” and “echo qvga-normal > /sys/bus/spi/devices/spi2.0/state”
    is needed for mplayer in 320×240 with correct colors.

  39. Timo Juhani Lindfors Says:

    What’s the status of getting this to mainline mplayer? Is this reasonably safe to use if one doesn’t run Xorg and reboots after playing the video before starting X again?

  40. Jonathan Says:

    Wow… took me 18 months of owning a moko to discover this. When it works it seems to work well. When it doesn’t I have to reboot the whole fr. So I’m trying to figure out how to make it work more reliably…

    Anyway I have two questions to add. FYI I’m running Debian using kernel 2.6.29-rc3 from SHR and X/xserver-xorg-video-glamo

    1. I get a permission problem if I don’t run mplayer as root. Any suggestion of which group I need to join my user to, or which files I need to change the permissions of?

    MPlayer dev-SVN-rUNKNOWN-4.1.2 (C) 2000-2008 MPlayer Team
    CPU: ARM

    Playing Transfer/insiders.mp4.
    AVI file format detected.
    [aviheader] Video stream found, -vid 0
    [aviheader] Audio stream found, -aid 1
    VIDEO: [FMP4] 320×180 24bpp 15.000 fps 304.6 kbps (37.2 kbyte/s)
    Clip info:
    Software: MEncoder SVN-r30656
    error opening mem: Permission denied
    VO XOverlay: Subvo init failed
    Error opening/initializing the selected video_out (-vo) device.

    2. It rejects some files that I have encoded as you suggest, eg the following is 320×180:

    ==========================================================================
    Opening video decoder: [hwmp4] MPEG-4 Video passthrough
    VDec: vo config request – 180 x 320 (preferred colorspace: MPEG-4)
    VDec: using MPEG-4 as output csp (no 0)
    Movie-Aspect is undefined – no prescaling applied.
    VO: [glamo] 180×320 => 180×320 MPEG-4
    bad image size

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


Follow

Get every new post delivered to your Inbox.

%d bloggers like this: