Open main menu

DAVE Developer's Wiki β

Changes

Deploying Embedded Linux Systems

7,893 bytes added, 10:58, 18 May 2023
Various
= Introduction =
Deployment of [[Embedded Linux]] systems is the typical operation which follows the development phase. When the application is ready and fully tested in the develoment development environment, it's time to take the system to the field for the “real work”. This phase brings a lot of concerns to cope with, for example creating a suitable root filesystem, saving the data properly, implement and implementing successful on-the-field update strategies. This how-to guide explains how to solve lists some of the problems connected common issues related to the deployment of an embedded linux Linux system.
= The development environment =
= Moving to the field =
When the system is ready to move to the field, most of the times the particular pysical link between host and target used at the development stage must be removed. In the worst case, the system must run without any NFS filesystem or file transfer services, relying only on its hardware resources (for example, the on-board ram RAM and flash memories). Resources that These resources are obviously limited, due to the nature of the embedded systems. Generally speaking, the procedure used to deploy the system configuration highly depends on the specific application; however, some topics are quite common. The following sections shine a light on these topics. ==Using an evaluation kit for the development stage==Often, it is convenient to carry out the development stage with an evaluation kit such as [[AXEL_ULite_and_SBC_Lynx_Embedded_Linux_Kit_(XUELK)|XUELK]] and [[Axel_Embedded_Linux_Kit_(XELK)|XELK]]. This is the case, for instance, when the final hardware platform is not available yet. It is worth remembering that, in this scenario, '''it is mandatory to port the software Board Support Package (BSP) to the final hardware platform when migrating from the evaluation kit'''. Conventionally, the BSP consists of:* Firmware (if any)* Bootloader (typically U-Boot) and, optionally, its device tree* Linux kernel and its device tree* Root file system.'''If not properly ported, the BSP may cause irreversible damage to the platform or severe malfunctioning as it is strictly related to the underlying hardware.''' Therefore, '''it is strongly discouraged to run the binary files released for DAVE Embedded Systems' evaluation kits on different platforms unless you know what you are doing.'''
= Root file systems =
Please note that init implementations can differ: Busybox offers its own version while other distributions, like Angstrom, can provide the classic System V version. More modern distros, like Ubuntu, implements Upstart, a complete replacement of the init daemon. For further information on the init process, please visit [http://www.yolinux.com/TUTORIALS/LinuxTutorialInitProcess.html this page].
 
= Non-volatile memories partitioning scheme =
In general, many different schemes can be implemented. Cost, reliability, speed are just some of the parameters to be considered when designing the partitioning scheme. [[Standalone_boot_(BELK/BXELK)|This article]] shows two quite different schemes on the same platform, for example.
 
= Startup sequence =
==Boot time==
Some applications require a quick boot time. When deploying an embedded Linux system, this time has to be evaluated in order to verify if it matches the system requirement, if any. Should the boot sequence is too long, several techniques can be used to speed it up. For more details, please see [[SDVX-TN-001:_Fast_boot_time|this page]] or [https://elinux.org/Boot_Time this article]. Generally speaking, it is worth to remember that '''different type of memories can lead to largely different boot times'''. A common scenario is the following:* The proof of concept (PoC) of a new product is developed using a microSD card as boot memory. Everything is stored there: bootloader, o.s. kernel, root file system, etc.*Then, the software is deployed onto the target according to the final production configuration by using SoM's onboard non-volatile memories. See [[Standalone_boot_(BELK/BXELK)#Use_case_.231|this real case]] as an example. Due to the fact that a large portion of the software is stored onto the NAND flash, the overall boot time may increase significantly. Why?**microSD card has a synchronous interface to the host, while raw NAND flashes used on DAVE Embedded Systems system-on-modules have not. Data communication between the system-on-chio (SoC) and the memory is much slower.**More importantly, raw NAND flashes are unmanaged memories. Unlike microSD cards, they don't embed a controller for handling the commands received from the host and taking care of low-level management of the underlying memory (which is based on NAND flash technology as well⁠). This includes but is not limited to wear leveling, bad blocks management, and read errors handling. In the case of raw NAND flashes, these tasks are all performed by the operating system running on the host processor.
= On-the-field software upgrades =
One of the greatest challenges for embedded systems manufacturers is to guarantee that the software on the system can be updated in the easiest way. On-the-field software upgradability is a major requirement that allows to replace bogus code and to enhance application features. How to perform this operation is highly platform-dependent. The following section shows in detail a specific situation.
 
As a general rule of thumb, '''it is highly recommended to avoid package-based systems such as rpm or deb to implement reliable software upgrade mechanisms.'''
== U-Boot/Linux system ==
More information is available in README, INSTALL and MULTI files included in the Dropbear distribution. Please note that for recent systems, as Lizard and Naon, Dropbear can be installed from '''pre-built packages''' (please refer to the distribution's package-manager documentation).
 
=Security=
Nowadays, a large number of newly designed embedded systems are used in Internet-connected products. As such, these systems have to face severe issues in terms of cybersecurity. This topic is huge and it is obviously beyond the scope of this document. It is worth remembering, however, that modern SOCs provide a rich set of native functionalities that address this issue. For instance, see [[XUELK-WP-001:_Secure_boot_on_iMX6UL|this page]] or [[BELK-TN-001:_Real-timeness,_system_integrity_and_TrustZone%C2%AE_technology_on_AMP_configuration|this technical note]]. These documents show how some of such functionalities were leveraged on DAVE Embedded Systems products.
 
For a more general discussion about the security of Internet-connected devices, see for example the [https://www.iotsecurityfoundation.org/ IoT Security foundation website] or [https://www.microsoft.com/en-us/research/publication/seven-properties-highly-secure-devices/ this paper] by Microsoft.
 
=Licensing=
It is known that a GNU/Linux system is based on a large amount of open source software. Consequently, the manufacturers of a product embedding such software must deal with the open source license compliance.
 
This topic is beyond the scope of this document, but a couple of links are provided anyway:
*[https://elinux.org/Legal_Issues This article] provides an overview of license-related legal issues
*Yocto build system generates automatically a manifest file to make the compliance management easier. For more details, please refer to [[https://www.yoctoproject.org/docs/1.8/dev-manual/dev-manual.html#maintaining-open-source-license-compliance-during-your-products-lifecycle|this link]]
 
=Certifications=
In general, all the products must conform to a set of standards in order to get the certificates required to be sold ([https://en.wikipedia.org/wiki/CE_marking CE marking] and [https://en.wikipedia.org/wiki/FCC_Declaration_of_Conformity FCC_Declaration_of_Conformity] are two well-known examples.
 
In many cases, software modifications to the Board Support Package (BSP) or the user applications are required to pass the compliance tests dictated by these regulations (*). In a typical scenario, these modifications are usually carried out after the development of the user applications. Nevertheless, these changes must be committed to the source repositories in order to be part of the production images that are stored into the product.
 
 
 
(*) A common example is the enabling of the spread-spectrum technique to reduce the electromagnetic emission peak associated with a digital clock, which is usually implemented at boot loader level.
=Misc=
'''DAVE Embedded Systems''' owns an IAB (Individual Address Block, a set of 4096 addresses), that is in the public listing, so everyone can find out that an address is associated to '''DAVE Embedded Systems'''. Note that the registration authority provides only IABs and OUIs (16000000+ addresses), and that a company is not allowed to request another IAB until at least 95% of the MAC addresses of the previous IAB have been used.
Customers who build their products using '''DAVE Embedded Systems'''' SOMs (NaonAXEL Lite, LizardAXEL ULite, QongBORA, ZefeerORCA,...) usually provide MAC numbers by themselves by acquiring them from IEEE. In fact, there are many reasons for this. Three can be stressed:
* A CPU module is NOT an end-product. It is not a product that goes directly to the final user as a LAN PCI board, or a printer server. So, in case of CPU modules, who gets a CPU module and build its own product with it, is responsible for handling the MAC address.
* Even if '''DAVE Embedded Systems''' programs the MAC address in flash (as an example) at the manufacturing stage, the customer may erase, overwrite, modify this number for the actual CPU module. Also, the strategy and the position (NOR, NAND, E2PROM,...) of the MAC address may vary. '''DAVE Embedded Systems''' cannot guarantee - in other words - that MAC address is maintained in the form and position it had when delivered.
* An end-product hosting a '''DAVE Embedded Systems''' CPU module is not always a '''DAVE Embedded Systems'''' product. When it is (and there are some examples), '''DAVE Embedded Systems''' puts the proper MAC address on the product. When it's not, DAVE can't provide MAC addresses: as already stated, the list of DAVE's MAC addresses is public, and by reading this list everybody can see that the product manufacturer is '''DAVE Embedded Systems''', which is not true.
 
==Handling different product models==
It is quite common to manufacture different product models on the top of the same hardware platform. In such cases, it is convenient to use a unified software for all models. DAVE Embedded Systems provides the [[ConfigID_and_UniqueID|ConfigID]] mechanism that can be exploited to achieve this goal. ConfigID can be exploited at boot loader level, at kernel level, and at application level.
 
 
= Deployment Checklist =
Here below, a list of topics that should be taken into account for Product design and deployment on the field.
 
This checklist tries to cover the most known project/product aspects which should be - ''at least'' - carefully understood and answered before a product is installed on the field.
 
== Non volatile storage ==
* Partitioning scheme
** Endurance/reliability: NOR/NAND/SD/eMMC evaluation respect to reliability, costs, write cycles, data retention, data dimension
** Redundant scheme or Recovery partition (A/B or A + recovery)
== Boot sequence ==
* boot time requirements (in terms of ''seconds'') and behaviours
* BSP boot time analisys
** u-boot, kernel, minimal rfs
** application startup
== Applications ==
* special framework required: Java, Qt, Javscript engines
== Accessibility ==
*serial console: availability (vs security)
** startup configuration: boot stop with special character
== Upgrade ==
* upgrade on the field
* Local or Remote
** with recovery or redundant partitions
*** tracking of upgrade failures for maintenance information
** On The Field: removable peripherals usage (SD, USB) and ''"non expert"'' upgrade
== Security ==
* "Security by design": take into account security requirements at the beginning of the project
* useful guideline formalized by da Microsoft for [https://azure.microsoft.com/it-it/blog/azure-sphere-is-here-public-preview-and-dev-kits-now-available Azure Sphere]
== Recovery and service ==
* complete automatic Disaster recovery
* complete - manually assisted - Disaster recovery
* partial Disaster recovery
* restore to Manufacturer configuration
* Service Mode
== Licensing==
* specific license requirements, i.e. Qt library
== Certification ==
* EMC certification
* other specific certification requests (or document requirement like MTBF, etc.)
== Various ==
* Watchdog
** in case of WDT intervention: tracking of watchdog event for software analysis
* MAC addresses handling: see [http://standards.ieee.org/develop/regauth/index.html IEEE registration authority]
* runtime automatic detection and configuration: possible usage of [[ConfigID_and_UniqueID | ConfigID]] technique
8,186
edits