So the community often shouts loudly about how long it takes for OTAs and the like. As CyanogenMod is now shipping on the Oppo N1, we thought we’d take a moment to show you what the process involves, and what the timelines look like. Please note that this is a representative example, not a hard rule in terms of timing, and we are likely ahead of the curve in comparison to others given our size and just having one device to deal with.
The issue: O-Click not working
So for this, we will use one of the actual bugs we received on the shipping phone – the version of the O-Click, the handy BT remote, made available at retail was newer than that of the pre-release hardware we developed against. A small difference in how the device presented itself introduced an incompatibility with our pairing code, rendering pairing broken (JIRA issue CMN-1, reported Jan 7th). Simple fix really, something we knocked out really quickly. So then the question becomes, how do we get it to the users who bought the device?
Incremental OTA system
Our installer users are already seeing the fruits of this labor, and we have been working on the groundwork since early 2013. Installer builds and those who purchased the CM N1 shipping device receive incremental updates, small fragments of updates that patch just the portions of CM that updated. We’re working on bringing this to a wider scale, but in the meantime, the installer allows us a controlled release stream of representative devices across various manufacturers, partition sizes and the like. We used the installer as the debut of this functionality because we know at all times what the latest build available in that release stream is; no need to worry about nightlies, or even of superfluous mods/hacks/addons, as those that stay on the installer builds are less likely to take on that sort of endeavor. The installer builds ship with a new package, named CMFota.apk, that handles the OTA download. Think of it as a stripped down version of CMUpdater, instead of configurably selecting updates, CMFota merely tells you when we’ve enabled another incremental for that device and flashes it.
The incremental system is the backbone for our future update and release plans. With a smaller delivery package, even those with data caps and bandwidth limitations can easily receive CM updates.
But back to the N1 example case – we’ve built the OTA system, we’ve packaged the bugfix, so its as simple as flipping a switch right? Not quite. Once the fix is implemented and packaged, the ota goes to Cyanogen Inc.’s QA team. That team is responsible for making sure the fix works, and doesn’t have any unintended consequences due to the ota system or other changes we may have since introduced. “Once QA is done, then you push the button?” Still no.
Once QA is satisfied with the OTA, we have to run the device through a series of tests known as CTS. This is a test suite that ensures that what we call ‘CyanogenMod’ the OS is still compatible with ‘Android’ as made available by Google and all the other OEMs. This is the backbone to Android compatibility regardless of OEM, making sure one app on CM functions identical to that same app on AOSP, Sense, TouchWiz and etc. CTS takes about 8 hours per execution, and is a somewhat fickle test, prone to issues. To pass, you need a 100% execution on a single run. Easy enough in practice, but any hiccup, even a network ping taking too long due to external circumstances, is a failure. You fail, you start the entire test again.
Once this test is passed, the results are blessed by the OEMs (and sometimes Google) and the OTA can officially be greenlit. Once greenlit, it is rolled out to users and they get the bug fixes included. Total time since the ‘fix’ was put in code, roughly 3 weeks with added time to due to our discussions with Google and/or Oppo as needed.
Extra time to get it right
If we wanted to fix a bug in say, the Nexus 5 installer builds, we could roll out the OTA in a day or two as we aren’t held to the CTS and other overhead constraints. But this isn’t a complaint of the process, the extra time spent to ensure everything goes off without a hitch is what the people who bought the device paid for. This is new territory for us, being the solely responsible entity for support, but it is a challenge that we enjoy having. The lessons learned will help future devices we ship, as well as directly assist with future updates of the Community Builds of CM.
I personally think its interesting to see the other side of the coin, so to speak. I hope you enjoyed this little look inside the ‘OEM” process.