Why should I care about dead brick?
Welcome to the world of embedded product development, any device running software needs firmware updates there is no denial on that. We use every channel to notify customers that they need to upgrade their device software asap. But why do customers panic when we serve them updates. There is no straight answer as each individual sees it differently. Common reasons are device seems to run slow, new features are not up to the mark, update is a time consuming process, device storage is going for a toss, new bugs could occur, new software consumes more battery power, dead brick!! Of all these dead brick is the scary thing which we find with most customers which prevents them from opting for firmware upgrades.
So What is this dead-brick – It is the situation where your device becomes unusable and unbootable after a firmware update, the only option to retrieve being shipping back to the factory.
At e-con our eSOMgears team treats dead brick very seriously for our OTA customers. Most customers are happy to live with their buggy devices than attempting firmware updates with dead brick nervousness. Putting this perception first is as important as designing a board that does not go up in flames because of heating effects. The reason for that is once a customers fully functional device goes for a toss after firmware updates it is as good as not owning that device. It really annoys people when the device is not use able. Especially in a production environment its even more damaging to the business and the mission at hand.
From experience we find that dead brick is not an easy problem to solve. It involves following good quality practices when building and deploying firmware to device run time monitors that trigger warnings and rollbacks. Let us give you some mechanisms we have in place to avoid dead brick when you use eSOMgears-OTA.
Level – 1 protection With eSOMgears-OTA firmware build kit one can build a firmware upload package. This firmware upload package will be uploaded to the eSOMgears cloud for device to download uploads. The build kit also has an option to build an SD card image. After a successful firmware build one can use the SD card image to test it on their development devices.
Level – 2 protection Once you upload this SD card tested image to the cloud, we by default does not make it available to devices in production, we push them to a category called test. What this means is you can have a host of devices configured under test running your previous version of software. The newly pushed firmware to the cloud is only available to these devices. One can go on to these devices and emulate the production environment. Each of these test devices could be configured as desired mimicking the production environment as much as possible.
A product QA team can thoroughly test and give it a go. Once the Service administrator gets all green he can switch the new firmware from test to production. Only when the firmware is switched from test to production the customer devices will receive update notifications.
eSOMgears users can manage and monitor all these from the dashboard analytics.
Level – 3 protection When you build your product with the device SDK and development kit that you download from eSOMgears cloud, we enable watchdogs on your device specially tailored for firmware updates. Whenever we see that a particular firmware update is blocking a successful boot up of the device in a given time frame we make best efforts to bring your device to the previous known good configuration. This way we make sure even if the firmware patches fail after your level 1 & 2 of testing in some devices on the customer premises we can still keep them functional. Such failures are reported to you through our dashboard, you can drill down and troubleshoot these special devices that reject your firmware updates.
In rare cases even these three levels of protection are not enough some products that demand high availability in which case we support customers with tailored solutions. On some devices where storage is not a premium and high availability is the key factor we can work with options like A/B updates.
With almost a decade and half of experience in embedded space we have created a highly usable firmware upgrade solution for our customers understanding the real needs. Our system has tracing mechanisms to troubleshoot issues for specific firmware upgrades which is not covered in this blog. The whole solution is highly managed, secure and reliable allowing to scale up only by adding more instances in the cloud. If you are in the embedded space, you surely need a firmware upgrade solution and you are in the right spot, we understand embedded very well.