MP4 improvements in Firefox for Android

One of the things that has always been a bit of a struggle in Firefox for Android is getting reliable video decoding for H264. For a couple of years, we've been shipping an implementation that went through great heroics in order to use libstagefright directly. While it does work fine in many cases, we consistently get reports of videos not playing, not displayed correctly, or just crashing.

In Android 4.1, Google added the MediaCodec class to the SDK. This provides a blessed interface to the underlying libstagefright API, so presumably it will be far more reliable. This summer, my intern Martin McDonough worked on adding a decoding backend in Firefox for Android that uses this class. I expected him to be able to get something that sort of worked by the end of the internship, but he totally shocked me by having video on the screen inside of two weeks. This included some time spent modifying our JNI bindings generator to work against the Android SDK. You can view Martin's intern presentation on Air Mozilla.

While the API for MediaCodec seems relatively straightforward, there are several details you need to get right or the whole thing falls apart. Martin constantly ran into problems where it would throw IllegalStateException for seemingly no valid reason. There was no error message or other explanation in the exception. This made development pretty frustrating, but he fought through it. It looks like Google has improved both the documentation and the error handling in the API as of Lollipop, so that's good to see.

As Martin wrapped up his internship he was working on handling the video frames as output by the decoder. Ideally you would get some kind of sane YUV variation, but this often is not the case. Qualcomm devices frequently output in their own proprietary format, OMX_QCOM_COLOR_FormatYUV420PackedSemiPlanar64x32Tile2m8ka. You'll notice this doesn't even appear in the list of possibilities according to MediaCodecInfo.CodecCapabilities. It does, however, appear in the OMX headers, along with a handful of other proprietary formats. Great, so Android has this mostly-nice class to decode video, but you can't do anything with the output? Yeah. Kinda. It turns out we actually have code to handle this format for B2G, because we run on QC hardware there, so this specific case had a possible solution. But maybe there is a better way?

I know from my work on supporting Flash on Android that we use a SurfaceTexture there to render video layers from the plugin. It worked really well most of the time. We can use that with MediaCodec too. With this output path we don't ever see the raw data; it goes straight into the Surface attached to the SurfaceTexture. You can then composite it with OpenGL and the crazy format conversions are done by the GPU. Pretty nice! I think handling all the different YUV conversions would've been a huge source of pain, so I was happy to eliminate that entire class of bugs. I imagine the GPU conversions are probably faster, too.

There is one problem with this. Sometimes we need to do something with the video other than composite it onto the screen with OpenGL. One common usage is to draw the video into a canvas (either 2D or WebGL). Now we have a problem, because the only way to get stuff out of the SurfaceTexture (and the attached Surface) is to draw it with OpenGL. Initially, my plan to handle this was to ask the compositor to draw this single SurfaceTexture separately into a temporary FBO, read it back, and give me those bits. It worked, but boy was it ugly. There has to be a better way, right? There is, but it's still not great. SurfaceTexture, as of Jelly Bean, allows you to attach and detach a GL context. Once attached, the updateTexImage() call updates whatever texture you attached. Detaching frees that texture, and makes the SurfaceTexture able to be attached to another texture (or GL context). My idea was to only attach the compositor to the SurfaceTexture while it was drawing it, and detach after. This would leave the SurfaceTexture able to be consumed by another GL context/texture. For doing the readback, we just attach to a context created specifically for this purpose on the main thread, blit the texture to a FBO, read the pixels, detach. Performance is not great, as glReadPixels() always seems to be slow on mobile GPUs, but it works. And it doesn't involve IPC to the compositor. I had to resort to a little hack to make some of this work well, though. Right now there is no way to create a SurfaceTexture in an initially detached state. You must always pass a texture in the constructor, so I pass 0 and then immediately call detachFromGLContext(). Pretty crappy, but it should be relatively safe. I filed an Android bug to request a no-arg constructor for SurfaceTexture more than two years ago, but nothing has happened. I'm not sure why Google even allows people to file stuff, honestly.

tl;dr: Video decoding should be much better in Firefox for Android as of today's Nightly if you are on Jelly Bean or higher. Please give it a try, especially if you've had problems in the past. Also, file bugs if you have issues!

Flash on Android 4.4 KitKat

There has been some some talk recently about the Flash situation on Android 4.4. While it's no secret that Adobe discontinued support for Flash on Android a while back, there are still a lot of folks using it on a daily basis. The Firefox for Android team consistently gets feedback about it, so it didn't take long to find out that things were amiss on KitKat.

I looked into the problem a few weeks ago in bug 935676, and found that some reserved functions were made virtual, breaking binary compatibility. I initially wanted to find a workaround that involved injecting the missing symbols, but that seems to be a bit of a dead end. I ended up making things work by unpacking the Flash APK with apktool, and modifying with a hex editor to replace the references to the missing symbols with something else. The functions in question aren't actually being called, so changing them to anything that exists works (I think I used pipe). It was necessary to pad each field with null characters to keep the size of the symbol table unchanged. After that I just repacked with apktool, installed, and everything seemed to work.

There is apparently an APK floating around that makes Flash work in other browsers on KitKat, but not Firefox. The above solution should allow someone to make an APK that works everywhere, so give it a shot if you are so inclined. I am not going to publish my own APK because of reasons.

Using direct textures on Android

I've been working at Mozilla on Firefox Mobile for a few months now. One of the goals of the new native UI is to have liquid smooth scrolling and panning at all times. Unsurprisingly, we do this by drawing into an OpenGL texture and moving it around on the screen. This is pretty fast until you run out of content in the texture and need to update it. Gecko runs in a separate thread and can draw to a buffer there without blocking us, but uploading that data into the texture is where problems arise. Right now we use just one very large texture (usually 2048x2048), and glTexSubImage2D can take anywhere from 25ms to 60ms. Given that our target is 60fps, we have about 16ms to draw a frame. This means we're guaranteed to miss at least one frame every time we upload, but likely more than that. What we need is a way of uploading texture data asynchronously (and preferably quicker). This is where direct textures can help.

If you haven't read Dianne Hackborn's recent posts on the Android graphics stack, you're missing out (part 1, part 2). The window compositing system she describes (called SurfaceFlinger) is particularly interesting because it is close to the problem we have in Firefox. One of the pieces Android uses to to draw windows is the gralloc module. As you may have guessed, gralloc is short for 'graphics alloc'. You can see the short and simple API for it here. Android has a wrapper class that encapsulates access to this called GraphicBuffer. It has an even nicer API, found here. Usage is very straightforward. Simply create the GraphicBuffer with whatever size and pixel format you need, lock it, write your bits, and unlock. One of the major wins here is that you can use the GraphicBuffer instance from any thread. So not only does this reduce a copy of your image, but it also means you can upload it without blocking the rendering loop!

To get it on the screen using OpenGL, you can create an EGLImageKHR from the GraphicBuffer and bind it to a texture:


GraphicBuffer* buffer = new GraphicBuffer(1024, 1024, PIXEL_FORMAT_RGB_565,
                                          GraphicBuffer::USAGE_SW_WRITE_OFTEN |

unsigned char* bits = NULL;
buffer->lock(GraphicBuffer::USAGE_SW_WRITE_OFTEN, (void**)&bits);

// Write bitmap data into 'bits' here


// Create the EGLImageKHR from the native buffer
EGLImageKHR img = eglCreateImageKHR(eglGetDisplay(EGL_DEFAULT_DISPLAY), EGL_NO_CONTEXT,

// Create GL texture, bind to GL_TEXTURE_2D, etc.

// Attach the EGLImage to whatever texture is bound to GL_TEXTURE_2D
glEGLImageTargetTexture2DOES(GL_TEXTURE_2D, img);

The resulting texture can be used as a regular one, with one caveat. Whenever you manipulate pixel data, the changes will be reflected on the screen immediately after unlock. You probably want to double buffer in order to avoid problems here.

If you've ever used the Android NDK, it won't be surprising that GraphicBuffer (or anything similar) doesn't exist there. In order to use any of this in your app you'll need to resort to dlopen hacks. It's a pretty depressing situation. Google uses this all over the OS, but doesn't seem to think that apps need a high performance API. But wait, it gets worse. Even after jumping through these hoops, some gralloc drivers don't allow regular apps to play ball. So far, testing indicates that this is the case on Adreno and Mali GPUs. Thankfully, PowerVR and Tegra allow it, which covers a fair number of devices.

With any luck, I'll land the patches that use this in Firefox Mobile today. The result should be a much smoother panning and zooming experience on devices where gralloc is allowed to work.

Android = Linux-on-anything-with-a-screen

Here we are in 2011. Obviously, the year of the Linux desktop, right? You'd hope so. It's not like there hasn't been plenty of time to make it happen. Red Hat, Canonical, Novell, and many other companies and individuals have been working hard on it for quite a while. And yet, very few inroads have been made. Sure, Dell has been selling Ubuntu on some netbooks with limited success, but that is hardly a victory.

In 2007, Google unveiled the Linux-based Android smartphone platform. The first phone wasn't released until a year later in 2008, along with the Android source code. Now, in Q4 2010, Android is the best-selling smartphone operating system in the world, surpassing Symbian, BlackBerry, Windows Mobile, and yes even iOS. In 2 years, it went from zero to total domination, with no signs of slowing down. Not everyone agrees that Android is the best smartphone OS (ahem, Gruber), but it's impossible to say that it isn't a success. And it uses Linux. In fact, I believe you could say that Android is the most successful and widely deployed Linux product ever. So why haven't we standardized on it for other non-smartphone uses? It would require some work, but at least there is some chance it could work out.

Initially I was going to try to call out all of the infuriating things about developing on desktop Linux, but I got tired. All of that has been said elsewhere by people smarter than me. The main reason I'm really behind Android is that it dramatically reduces the amount of fragmentation. It's just not possible for a hardware vendor like AMD or NVIDIA to support the multitude of distros out there. If you told those guys they only had to support a couple versions of Android, they would be able to deliver much higher quality of support.

The Eclipse IDE, Dalvik VM, and Android application framework are all great assets and deliver a far superior development experience over standard Linux. You get that for free. If Java isn't your bag, though, you can always use the Native Development Kit to write in C, C++, C#, or whatever you want. I would love to see Clutter on Android, for instance. And Node.js/gjs. Or any number of other great projects.

It's already possible to run Android on a PC via the Android-x86 project. They've added basic mouse support, as well as hardware accelerated OpenGL ES2 via Mesa (on Intel hardware). I have a Cr-48 running it here, and it works amazingly well. Even wifi works.

Yeah, I'm a Rails fanboy now

A lot of my day job now involves working with Ruby on Rails. At first I wasn’t sure how much I would like Rails or ruby, given that I had been doing a lot of C#/C/whatever desktop work before. Not surprisingly, though, I’ve become quite addicted. The test-driven nature of development is a welcome change — most desktop apps I worked on didn’t even have tests. The Rails community has done a great job of banging automated testing into people’s heads. Almost every tutorial, book, or random blog post I’ve seen emphasizes the importance of good automated tests. Hopefully it has helped decrease the instances of ‘snorpage’, but perhaps my co-workers would disagree :)

Anyway, I love Rails so much that I’ve converted my blog from Wordpress to Enki. Enki is more of a create-your-own-blog construction kit than a turn-key solution like Wordpress. That was one of the main reasons I chose it over Typo or Mephisto — I wanted to be able to easily hack on it.

I wrote a quick and dirty script to help me import the Wordpress posts into Enki. Any fellow Enki hackers can grab it here.