Mostly all of today's mobile operating systems make it easy for the end user to keep a smartphone up-to-date to the latest firmware version and either handset manufactures, operating system providers or carriers can offer new features through firmware updates or simply fix bugs from the previous version. Furthermore, firmware updates allow upgrades to the next operating system version without forcing the user to buy a new handset. However, in the early days of mobile phones, devices haven't received too many updates at all and customers weren't too keen to update their devices, especially since these phones weren't smartphones at all and there was no real reason to update the firmware anyway.
However, even in the early days there was one exception which was about getting serious bugs fixed but in most cases this update had to happen at service centers and it didn't happen too often because mobile phones were more simple most handsets were rocking solid engineered anyway. However, in today's fast smartphone world with new Internet services coming up every day, firmware updates became quite common and its seems that more and more reports hit news sites and forums that firmware updates failed badly, bricked devices and carriers as well as manufactures are taking back updates, just to offer a new, better version the next day.
Because of many misunderstandings, wrong assumptions and speculations, yesterday we took the time to sit down with O2 Germany's handset product management to talk about how updates are planned, developed and rolled-out to finally demystify the whole story and to clean-up with widely used misinformation.
A firmware update, doesn't matter if it's rolled out by a handset manufacture only (the so called Open Market version) or rolled out by handset manufactures and carriers together (also called Customized or Carrier version) always contains five steps:
- Development / ROM Building
- Acceptance / Approval
- Roll-Out / Distribution
As soon as a new release is available for a certain handset (with bug fixes only or feature updates or even version upgrades), the handset manufacturer either plans to make this version available or not.
- a) If it's planned to update the Open Market version of a certain handset, the manufacturer starts with the Development / ROM Building (2).
- b) If a manufacture plans to update the Open Market version, it's also offered to carrier customers and the carrier is informed that a new version is available and both, the manufacturer and the carrier either agree to roll it out or wait for the next version. If it's agreed to roll-out a new firmware version commercially, the Development / ROM Building process starts (2).
In any case, the manufacturer and the carrier agree together for carrier specific versions, if a firmware update will be offered or not. However, quite often carriers also request firmware updates for certain handsets which might not planned to be updated through the Open Market version and in this case it's up to the manufacturer to either offer it or not. Quite often, a device is so called "end of life" already and the engineers are working on new devices already or a device is technically not able to support or run a new firmware version (which often happens to version upgrades).
2. Development / ROM Building
The manufacturer puts together all required bits and bytes, received from the operating system provider (often called "Golden Code" or "Ready To Manufacturing (RTM)") and adds its own pieces of code which is required for a certain device, like drivers, etc.
- a) If the firmware goes Open Market, the manufacturers may also add a wide range of standard carrier settings, like APN and MMS settings, by themselves. As soon as the new ROM build is done, it will goes to Testing (3).
- b) If the firmware is a carrier specific version it’s nevertheless mostly based on the Open Market version anyway. But the carrier might request to add special settings and services like partner settings or services like bookmarks or further applications. After the firmware is built, it goes to Testing (3).
In any case, the handset manufacture is building the firmware, not the carrier and in most cases a carrier version is always build as a new version of the original Open Market version. However, the manufacturer relies on information and input it gets from the carrier, like APN and MMS settings and 3rd party applications, which might be added to the ROM build. Nevertheless, only the manufacturer takes care that the whole firmware, with the operating system, radio stack, drivers, etc. is working well. Carriers can only give input but aren't developing anything for ROM builds except maybe 3rd party applications.
Doesn't matter if a firmware is Open Market only or carrier specific, the manufacturer is testing the ROM builds first.
- a) If an Open Market build passes all the internal tests, it might be handed over to the operating system provider (for instance Google or Microsoft) for another test and Approval (4). Otherwise, if it fails the internal tests and requires a rework, it goes back to Development / ROM Building (2).
- b) If the firmware is carrier specific, after the internal tests, it's handed-over to the carrier for further network tests. This is an important part of the whole testing and customer satisfaction process because only carriers can test certain handsets and ROM builds or device features against their own infrastructure. If the firmware passes all carrier tests and receives the carrier acceptance, it's reported back to the manufacturer which might hand it over to the operating system provider (for instance Google or Microsoft) for another test and Approval (4). Otherwise, if the carrier isn't satisfied with the firmware quality or if the carrier finds major or minor issues which requires a rework, the build goes back to the Development / ROM Building (2) again.
While handset manufactures have a lot of carrier specific information today, and therefore can self-certify a firmware build that it is compliant to the carrier's infrastructure, the carrier test is an important part of the whole development process. Only carriers can test and adjust network related features like Visual Voicemail or MMSC settings which offer the end user the best possible experience.
While ROM updates often require so called sanity tests of 100 - 500 test cases only, brand new devices or major software releases often run through 3000 or more test cases!
No question, this is a time consuming process which adds some extra days to the whole development process but testing and quality control is often an underestimated task until a deployed firmware fails to work proper.
4. Acceptance / Approval
Basically the whole acceptance and approval process includes two or three parties: The manufacturer, the carrier (only if it's a carrier firmware) and the operating system provider.
- a) While the manufacturer takes care that all hardware and software components work together well, the operating system provider takes care that its own services work sufficient as well as branding guidelines might be respected. If both, the manufacturer and the operating system provider are satisfied with the firmware build, it’s accepted and approved for the Roll-Out / Distribution (5).
- b) If the firmware version is a carrier version it furthermore needs also to be approved by the network operator since the carrier takes care that the firmware works with the network and supports all offered features. If also accepted by the carrier, the firmware is ready for Roll-Out / Distribution (5).
This approval process became pretty tricky in the smartphone days. While in the early mobile phone days only the manufacturers and the carriers were involved into the acceptance and approval process, now there's also the operating system provider involved because it's not about simple operating systems anymore but also about services and brand. For instance Google takes care that the Google experience with Gmail, Maps, Android Market, etc. is represented on the device as Google designed it. And the same will happen with Windows Phone 7 devices where Microsoft will also test services like Zune or Bing as well as Microsoft will take care that Hubs are represented the way they were designed.
It can happen that a manufacturer as well as a carrier is happy with a build quality but the operating system provider isn't and everything starts again from Development / Rom Building (2). Furthermore, operating system providers have to test devices from plenty of manufacturers and even more carriers from around the world which can take its time, even if these kinds of processes are highly optimized today.
5. Roll-Out / Distribution
After the firmware build were approved by all involved parties it can be rolled-out and distributed to the end user.
a) If it's an Open Market version, the manufacturer is rolling it out without any further notice.
b) If the firmware is a Customized carrier version, the manufacturer informs the carrier that the ROM build was approved and that it is now ready for distribution. Both, the manufacturer and the carrier agrees on a day of release which gives the carrier further time to inform its customer care, that a new version is rolled-out soon.
Firmware version can be distributed in two different ways or in parallel: Side Load (using a PC and a cable) and FOTA (Firmware Over The Air) which both can either include a full firmware update (in most cases > 100 MB) or just an incremental update where only certain parts of the current device's firmware are replaced with new bits and bytes (in most cases several MB only). If it's a complete firmware update, in most cases all stored information is overwritten by the new firmware and all data, including installed applications, is lost. If it's just an incremental update, in most cases no information or data is lost at all.
While Side Load is typically used by Apple's iPhone and was so far used by Microsoft's Windows Mobile/Windows Phone smartphones, FOTA is the only way to upgrade Palm's webOS devices and the preferred method for Android and the upcoming Windows Phone 7 smartphones, even if both - Android and Windows Phone 7 - are/will also support Side Load updates.
- Side Load: Either the carrier or the manufacturer hosts the firmware update on its own servers from where the firmware can be either downloaded through a web site or from where it is automatically downloaded through a PC software. For instance, in the past, Windows Mobile updates where directly downloaded through a web browser as a full software package which also included the required flash utility. Nowadays, PC suites like Samsung's New PC Suite or Sony Ericsson's PC Companion can also flash devices and check the web servers periodically if a new firmware version is available.
- FOTA: Firmware Over The Air updates are also either hosted by manufactures or carriers and the software is directly distributed to the handset from where it is installed to. In most cases, the device notifies the user that a firmware update is available, and the user can decide to either download via WiFi or WWAN, immediately or later. Furthermore, if a FOTA update is offered, many carriers and handset manufactures also offer a Side Load download as an alternative or as a fallback solution.
The distribution process and device update can be somewhat tricky since this is the time the new firmware is installed on the customer's mobile device where the previous firmware is either totally or partly replaced. If anything goes wrong here or any drivers are missed or anything isn't installed correctly, the update can brick the device. While this doesn't mean that the device past remedy, from the customers point of view it's the most critical part of the whole update process.
Done, the new firmware is delivered and installed on the end users mobile phone and the process starts from the beginning as soon as a new version is decided to be released.
The steps above sounds pretty straight forward and in theory it is. However, smartphone software became a very complex topic over the years and all involved parties pay the most attention to not deliver any faulty firmware updates. However, it can happen and there are some simple reasons why it can happen. For instance for Android and Windows Mobile/Windows Phone in most cases the full ROM updates are tested, by simply side loading them to test devices even if an incremental update is delivered later to the end users. It's simply done to minimize the testing process and therefore reducing the time to delivery. Otherwise, the carrier's or manufacturer's update server needs to be configured to distribute test ROM updates to a predefined range of test devices only. This is a very time consuming and expensive process and in most cases not really necessary at all. However, even if it's not the standard, from time to time this additional scenario is also tested. The only exception is webOS since Palm is only offering FOTA updates and therefore all test builds are distributed over the air as well.
So if an update isn't planned exactly, if can happen that the full update contains all the required bits, bytes and drivers, but the incremental update might miss the one or the other important driver which is required for certain handsets which might use different components now than previous versions of the same handset (think about AMOLED vs. SLCD display changes or changed memory or radio components). If such an incremental update is now installed on a device with a different hardware set-up, the update will fail and often brick the device, even if the full update was tested by three different instances successfully. This is one of the major reasons, why some updates might fail to work.
Nevertheless, and this is important to keep in mind: Only the handset manufacturer is in charge for building firmware updates. The operating system provider has done its work before, when the golden code was released to the manufacturer and carriers don't develop or build firmware updates at all. They might let the manufacturer add own applications to the complete firmware package but these packages are build by the handset manufacturers only.
Furthermore, carriers aren't wasting anyone's time with extra tests and acceptance processes. All this is very expensive for carriers and they don't do it to bother its customers which might urgently await the firmware update, but carriers are doing it to make sure the device is providing the best possible experience. These kinds of testing often include scenarios like handovers from 2G to 3G networks and back, international roaming behaviors and the minimization of battery draining. So all this happens to the benefit of the end users and we are talking about millions of end users a carrier has to take care about. The über-geeks, which would like to get the firmware update better yesterday than tomorrow are the absolute minority!
All together, rolling-out a new firmware for an existing device is a pretty complex and time consuming project.
Cheers ~ Arne