A case study: Oppo N1, Bugfixes and OTAs

So the community often shouts loudly about how long it takes for OTAs and the like. As CyanogenMod is now shipping on the Oppo N1, we thought we’d take a moment to show you what the process involves, and what the timelines look like. Please note that this is a representative example, not a hard rule in terms of timing, and we are likely ahead of the curve in comparison to others given our size and just having one device to deal with.

The issue: O-Click not working

So for this, we will use one of the actual bugs we received on the shipping phone – the version of the O-Click, the handy BT remote, made available at retail was newer than that of the pre-release hardware we developed against. A small difference in how the device presented itself introduced an incompatibility with our pairing code, rendering pairing broken (JIRA issue CMN-1, reported Jan 7th). Simple fix really, something we knocked out really quickly. So then the question becomes, how do we get it to the users who bought the device?

Incremental OTA system

Our installer users are already seeing the fruits of this labor, and we have been working on the groundwork since early 2013. Installer builds and those who purchased the CM N1 shipping device receive incremental updates, small fragments of updates that patch just the portions of CM that updated. We’re working on bringing this to a wider scale, but in the meantime, the installer allows us a controlled release stream of representative devices across various manufacturers, partition sizes and the like. We used the installer as the debut of this functionality because we know at all times what the latest build available in that release stream is; no need to worry about nightlies, or even of superfluous mods/hacks/addons, as those that stay on the installer builds are less likely to take on that sort of endeavor. The installer builds ship with a new package, named CMFota.apk, that handles the OTA download. Think of it as a stripped down version of CMUpdater, instead of configurably selecting updates, CMFota merely tells you when we’ve enabled another incremental for that device and flashes it.

The incremental system is the backbone for our future update and release plans. With a smaller delivery package, even those with data caps and bandwidth limitations can easily receive CM updates.

But back to the N1 example case – we’ve built the OTA system, we’ve packaged the bugfix, so its as simple as flipping a switch right? Not quite. Once the fix is implemented and packaged, the ota goes to Cyanogen Inc.’s QA team. That team is responsible for making sure the fix works, and doesn’t have any unintended consequences due to the ota system or other changes we may have since introduced. “Once QA is done, then you push the button?” Still no.

Once QA is satisfied with the OTA, we have to run the device through a series of tests known as CTS. This is a test suite that ensures that what we call ‘CyanogenMod’ the OS is still compatible with ‘Android’ as made available by Google and all the other OEMs. This is the backbone to Android compatibility regardless of OEM, making sure one app on CM functions identical to that same app on AOSP, Sense, TouchWiz and etc. CTS takes about 8 hours per execution, and is a somewhat fickle test, prone to issues. To pass, you need a 100% execution on a single run. Easy enough in practice, but any hiccup, even a network ping taking too long due to external circumstances, is a failure. You fail, you start the entire test again.

Once this test is passed, the results are blessed by the OEMs (and sometimes Google) and the OTA can officially be greenlit. Once greenlit, it is rolled out to users and they get the bug fixes included. Total time since the ‘fix’ was put in code, roughly 3 weeks with added time to due to our discussions with Google and/or Oppo as needed.

Extra time to get it right

If we wanted to fix a bug in say, the Nexus 5 installer builds, we could roll out the OTA in a day or two as we aren’t held to the CTS and other overhead constraints. But this isn’t a complaint of the process, the extra time spent to ensure everything goes off without a hitch is what the people who bought the device paid for. This is new territory for us, being the solely responsible entity for support, but it is a challenge that we enjoy having. The lessons learned will help future devices we ship, as well as directly assist with future updates of the Community Builds of CM.

I personally think its interesting to see the other side of the coin, so to speak. I hope you enjoyed this little look inside the ‘OEM” process.

  • blue presto

    I posted the CMN-1 after waiting a while and prodding Steve to get a Jira section posted. In the future it would be fantastic if you could either A. Respond to or assign the Jira… That way we know you are aware of the issue. B. Have a section in the CM forums where you let us know you are aware of major problems. C. Have a section in Oppo forums where you respond. D. Issue some sort of brief status report here.
    A bunch of CM enthusiasts were pushed into Omni ROM because of the very long communications black out. The lesson here is at least give a quick note saying you are aware of the problems.

  • Ronaldo Medeiros

    Too late for explanations… Keep us in the dark for too long… I loved you CM, but…

    i9100 with OmniROM 4.4 Kitkat now! running perfectly

  • blue presto

    And saying reported Jan 7 was an unnecessary jab. It was reported promptly in the Oppo forums in December. There was no section in Jira for N1 until the 7th.

  • http://www.respond.hu/ Endless Nameless

    Sarcasm. I hope.

  • Eric Finkenbiner

    Some valid criticisms here in the comments, but I’m inclined to cut the team some slack on this one. I really appreciate the blog post though and truly love my N1.

  • Zachary Hinson

    I really wanted a m1 cm11 of hercules. What happened..

  • cronie

    Would it not be possible to allow users to skip the CTS test with a pre-release option in CMFota? If the QA team is satisfied, allow installation and create a backup of the latest CTS-approved files. If the fix changes or the pre-release option is disabled, roll it back to the latest CTS backup and then install the new updates if necessary

  • Nick Egerton

    As a general note from this debacle; maybe Cyanogen, Inc. can take the comments here as a lesson. Cyanogen’s communication in general, frankly, just sucks. For my device in particular (Sprint HTC One) there hasn’t been a stable or ‘M’ release since early November. I get that development isn’t easy, and this delay is totally acceptable and reasonable. But a heads up to your community (now your customers) would go a long way. My current assumption is that there simply may not be another stable build for my device.

    It’s hard to manage customers’ expectations, especially when you have so many different versions of your product, each essentially creating another separate community. But it is also your responsibility as a company.

    Case in Point: HTC’s Kit Kat update progress communication has been far, far clearer and transparent than Cyanogen’s. That should be a wake up call.

  • AmpVee

    I like how people are announcing they’re jumping ship at the slightest sign of inconvienence. Nice job by the way. Community spirit at work. I have yet to see x86 CM builds, but I’m holding strong.

  • http://www.aquademica.se/besiktning/vattenskada/ vattenskada

    Interesting to see … thank you it’s well done :)

  • Tadeusz Rutecki

    What’s wrong with you, as there is something wrong, then write!!

  • SlowMod

    Please do kitcat for Samsung Galaxy Note 2

  • SlowMod

    Kitkat for Galaxy Note 2

  • SlowMod

    Please

  • Bingster

    for i1900 users. go and get cyan fox rom 4.4.2 kitty kat…. it is very smooth :)

  • https://www.fixmycomputerdude.com/ Aliza Carvor

    I read whole the case post and get more inform about the OEM process thanks for the sharing the information.

  • Bingster

    Ho can anyone expect community spirit, when we have been kept in the dark for god knows how long. I have moved on to CyanFox for my i1900 and it is amazing. Cu

  • Jordan Pt

    Yeah my Nook hasn’t gotten a single M release since Nov.1 and I can’t find a single post anywhere talking about why things are held up. I’m sure some post in the Dev threads on XDA explain it but there are 20-50 pages each thread of posts that I can’t just sit and read.

  • Nick Egerton

    Given cyanogen’s community roots, it is genuinely ironic. I also sent an email and heard nothing back after weeks. How can they not put their best foot forward?

  • Tadeusz Rutecki

    where the window on Friday?

  • squiddy20

    While this has nothing to do with the N1, it would also be great if the CM team/maintainers would communicate with each other. On CM 11 (M, Nightly, and even one or 2 of the Installer builds) for toro/toroplus, there was/is a (admittedly minor) graphical issue characterized by tiny little vertical black lines most notably in the app switcher and when the screen rotates. This issue was apparently fixed in toro several weeks ago, but is still present in toroplus. When asked by many other toroplus users on the CM forums about why it hasn’t been fixed, the most common response was/is something to the effect of “the maintainers don’t usually come here”. Given that this website and these forums are dedicated only entirely to CM, I don’t see why this is the case. A lot of people (myself included) didn’t even know who the toro/toroplus maintainers were (and while the Wiki list was updated Jan 26, the person listed as maintainer for toroplus hasn’t been in that position for several months).

  • Oliver Schaub

    You are aware, that this is the OFFICIAL ROM for the Oppo-N1 we are talking about here, right?