Changes

Jump to: navigation, search

Deploying Embedded Linux Systems

8,958 bytes added, 07:16, 17 September 2019
Boot time
= Introduction =
Deployment of [[Embedded Linux]] systems is the typical operation which follows the development phase. When the application is ready and fully tested in the develoment development environment, it's time to take the system to the field for the “real work”. This phase brings a lot of concerns to cope with, for example creating a suitable root filesystem, saving the data properly, implement and implementing successful on-the-field update strategies. This how-to guide explains how to solve lists some of the problems connected common issues related to the deployment of an embedded linux Linux system.
= The development environment =
[[File:Development env.png]]
The host (usually a PC or a virtual machine running the Linux operating system) is used by the developer to (cross-)compile the code that will run on the target, for example a Dave '''DAVE Embedded Systems''' ARM CPU module such as Lizard or Naon. The Linux kernel running on the target is able to mount the root file system from different physical media. During the software development, it is very common to use a directory exported via [[W:Network File System|NFS]] by the host for this purpose. Moreover, the linux kernel is usually retrieved by a simple network transfer protocol like [http://en.wikipedia.org/wiki/Trivial_File_Transfer_Protocol tftp].
= Moving to the field =
# start from a small file system and add all the components (packages, libraries, application binaries, ..) that you need
Option #2 is always preferrable, because it leads to a very space-optimized root file system, but it could be more demanding, especially when you need to save just little storage space compared to the size of the original RFS (in this case, you can easily go for Option #1).
Please see the [[Embedded_Linux#Embedded distributions|Embedded distros]] article for an introduction on Embedded Linux distributions.
Please also refer to the following articles for additional information:
 
* http://www.denx.de/wiki/view/DULG/RootFileSystemDesignAndBuilding
If you prefer to build the entire root file system, there are several possibilities that are described in the following sections.
Yocto is sponsored by the [http://www.linuxfoundation.org/ Linux Foundation]
 
At the time of this writing (May 2019), '''Yocto has become by far the most popular build system in the Linux embedded arena'''. Large silicon vendors such as NXP, Texas Instruments, and Xilinx base their BSPs on Yocto. That's why '''all the Linux development kits released by DAVE Embedded Systems are in turn based on Yocto for some years now'''.
=== Arago ===
=== Buildroot ===
[http://buildroot.uclibc.org/ Buildroot] is a set of scripts and patches for the creation of a cross-compilation chain toolchain as well as the creation of a complete root file system.
=== Linux From Scratch ===
This step is clearly required to add to the basic root file system your custom application files (libraries, binaries, configuration files, ...)
=== Adding libraries Application dependancies ===
The applications executables that you have developed might depend on libraries that are not provided by the basic root file system. In this case these libraries must be added. To find out which libraries your applications depends on you can use the <code>ldd</code> and <code>readelf</code> tools. Please note that often you'll need to rebuild some libraries, cross-compiling them to match your target architecture.
Please note that init implementations can differ: Busybox offers its own version while other distributions, like Angstrom, can provide the classic System V version. More modern distros, like Ubuntu, implements Upstart, a complete replacement of the init daemon. For further information on the init process, please visit [http://www.yolinux.com/TUTORIALS/LinuxTutorialInitProcess.html this page].
=Non-volatile memories partitioning scheme = Deploying In general, many different schemes can be implemented. Cost, reliability, speed are just some of the root file system == = Watchdog =parameters to be considered when designing the partitioning scheme. [[Standalone_boot_(BELK/BXELK)|This article]] shows two quite different schemes on the same platform, for example.
= Startup sequence =
== The serial port ==U-Boot implements a text console on the serial port. This console can be used to stop Configuring the startup sequence to allow an interactive session with the human operator. This is very useful during debug operations, but by default any character the bootloader receives during the startup sequence stops this process. As a side effect, any spurious character received from the serial port devoted to the console is able to prevent the bootloader to complete the automatic boot process (and typically to start the operating system).  loader== Autoboot configuration ==For this reason before moving to the field it is highly recommended to configure the bootloader to halt the sequence when receiving a specific string. Fortunately, Autoboot process is deeply configurable: parameters defining the retry behaviour and the strings used to stop booting can be specified. Please read the README.autoboot file, provided inside the documentation directory of the U-Boot sources, for more details. = Setting the MAC address = In case the system provides Ethernet interface, it must be guaranteed that each device is delivered with a unique MAC address. MAC addresses are managed by IEEE Registration Authority: IEEE Registration Authority IEEE Standards Department 445 Hoes Lane Piscataway NJ 08854 Phone: (732) 562-3813 Fax: (732) 562-1571 http://standards.ieee.org/contact/form.html
For more details see also:<br>=== Handling the serial port ===* http://standardsU-Boot implements a text console on the serial port.ieeeThis console can be used to stop the startup sequence to allow an interactive session with the human operator.org/develop/regauth/indexThis is very useful during debug operations, but by default, any character the bootloader receives during the startup sequence stops this process.html* http://standards.ieee.org/develop/regauth/tut/indexAs a side effect, any spurious character received from the serial port devoted to the console is able to prevent the bootloader to complete the automatic boot process (and typically to start the operating system).html
Dave owns an IAB (Individual Address BlockFor this reason, before moving to the field it is highly recommended to configure the bootloader to halt the sequence when receiving a set of 4096 addresses)specific string instead. Fortunately, that Autoboot process is in deeply configurable: parameters defining the retry behaviour and the public listing, so everyone strings used to stop booting can find out that an address is associated to Davebe specified. Note that Please read the registration authority provides only IABs and OUIs (16000000+ addresses)README.autoboot file, and that a company is not allowed to request another IAB until at least 95% of provided inside the MAC addresses documentation directory of the previous IAB have been usedU-Boot sources, for more details.
Customers who build their products around DAVE's processor modules (Naon, Lizard, Qong, Zefeer,...) usually provide MAC numbers by themselves by acquiring them from IEEE. In fact there are many reasons for that. Three can be stressed:===Automating operations at boot loader level=== * A CPU module is NOT Modern boot loaders such as U-Boot allows automating boot operations in an end-productextremely flexible fashion. It is not a product that goes directly Thanks to the final user as a LAN PCI board, or a printer server. So, in case of CPU modulesscripting capabilities, who gets a CPU module and build its own product with it, is responsible for handling the MAC addresspossible to execute complex scripts.* Even if DAVE programs the MAC address in flash (as an example) at manufacturing stageFor more details, customer may erase, overwrite, modify this number see for the actual CPU moduleinstance [https://elinux. Also, the strategy and the position (NOR, NAND, E2PROM,org/ECE597_boot.scr this tutorial] or [https://elinux.org/ECE597_boot.) of the MAC address may vary. DAVE cannot guarantee - in other words - that MAC address is maintained in the form and position it had when delivered.* An end-product hosting a DAVE's CPU module is not always a DAVE's product. When it is (and there are some examples), DAVE puts the proper MAC address on the product. When it's not, DAVE can't provide MAC addresses: as already stated, the list of DAVE's MAC addresses is public, and reading scr this list someone can see that the product manufacturer is DAVE, which is not truepage].
==Boot time==
Some applications require a quick boot time. When deploying an embedded Linux system, this time has to be evaluated in order to verify if it matches the system requirement, if any. Should the boot sequence is too long, several techniques can be used to speed it up. For more details, please see [[SDVX-TN-001:_Fast_boot_time|this page]] or [https://elinux.org/Boot_Time this article].
Generally speaking, it is worth to remember that '''different type of memories can lead to largely different boot times'''. A common scenario is the following:
* The proof of concept (PoC) of a new product is developed using a microSD card as boot memory. Everything is stored there: bootloader, o.s. kernel, root file system, etc.
*Then, the software is deployed onto the target according to the final production configuration by using SoM's onboard non-volatile memories. See [[Standalone_boot_(BELK/BXELK)#Use_case_.231|this real case]] as an example. Due to the fact that a large portion of the software is stored onto the NAND flash, the overall boot time may increase significantly. Why?
**microSD card has a synchronous interface to the host, while raw NAND flashes used on DAVE Embedded Systems system-on-modules have not. Data communication between the system-on-chio (SoC) and the memory is much slower.
**More importantly, raw NAND flashes are unmanaged memories. Unlike microSD cards, they don't embed a controller for handling the commands received from the host and taking care of low-level management of the underlying memory (which is based on NAND flash technology as well⁠). This includes but is not limited to wear leveling, bad blocks management, and read errors handling. In the case of raw NAND flashes, these tasks are all performed by the operating system running on the host processor.
= On-the-field software upgrades =
* /dev/mtd5 -> additional file system
Basically , we can upgrade the system through the bootloader or through the kernel.
Finally, we assume that only the aforementioned components should be upgraded. In case the system equips external '''microcontrollers, FPGAs, CPLDs,''' etc, different strategies must be taken into account depending on the particular case.
=== Local upgrading ===
When the system doesn't allow a remote access, an operator must locally access the system through its user interface (in the best case) or through the serial port (in the worst case). If the
system provides a USB port, SD, MMC or PCMCIA slot, the system can retrieve the upgrade packages from those memory devices; if the user interface features a sort of upgrade command, the operator should just plug the device and activate the upgrade function. The serial port is the last option to connect to the system and manually send the proper commands to complete the upgrade procedure.
* Run the make tool <pre>make PROGRAMS=”dropbear dbclient scp” MULTI=1 STATIC=1</pre>
More information are is available in README, INSTALL and MULTI files included in the Dropbear distribution. Please note that for recent systems, as Lizard and Naon, Dropbear can be installed from '''pre-built packages''' (please refer to the Angstrom distribution's package-manager documentation). =Security=Nowadays, a large number of newly designed embedded systems are used in Internet-connected products. As such, these systems have to face severe issues in terms of cybersecurity. This topic is huge and it is obviously beyond the scope of this document. It is worth remembering, however, that modern SOCs provide a rich set of native functionalities that address this issue. For instance, see [[XUELK-WP-001:_Secure_boot_on_iMX6UL|this page]] or [[BELK-TN-001:_Real-timeness,_system_integrity_and_TrustZone%C2%AE_technology_on_AMP_configuration|this technical note]]. These documents show how some of such functionalities were leveraged on DAVE Embedded Systems products. For a more general discussion about the security of Internet-connected devices, see for example the [https://www.iotsecurityfoundation.org/ IoT Security foundation website] or [https://www.microsoft.com/en-us/research/publication/seven-properties-highly-secure-devices/ this paper] by Microsoft. =Licensing=It is known that a GNU/Linux system is based on a large amount of open source software. Consequently, the manufacturers of a product embedding such software must deal with the open source license compliance. This topic is beyond the scope of this document, but a couple of links are provided anyway:*[https://elinux.org/Legal_Issues This article] provides an overview of license-related legal issues*Yocto build system generates automatically a manifest file to make the compliance management easier. For more details, please refer to [[https://www.yoctoproject.org/docs/1.8/dev-manual/dev-manual.html#maintaining-open-source-license-compliance-during-your-products-lifecycle|this link]] =Certifications=In general, all the products must conform to a set of standards in order to get the certificates required to be sold ([https://en.wikipedia.org/wiki/CE_marking CE marking] and [https://en.wikipedia.org/wiki/FCC_Declaration_of_Conformity FCC_Declaration_of_Conformity] are two well-known examples. In many cases, software modifications to the Board Support Package (BSP) or the user applications are required to pass the compliance tests dictated by these regulations (*). In a typical scenario, these modifications are usually carried out after the development of the user applications. Nevertheless, these changes must be committed to the source repositories in order to be part of the production images that are stored into the product.   (*) A common example is the enabling of the spread-spectrum technique to reduce the electromagnetic emission peak associated with a digital clock, which is usually implemented at boot loader level. =Misc=== Watchdog == Typically, during application development, the watchdog device included in the embedded system is turned off. Before moving to the field, enabling the watchdog is often a mandatory choice (*). The use of this peripheral is a little bit tricky because it involves both U-Boot and Linux. The following sequence shows the typical scenario when the system is working on the field:  # Processor comes out of reset; internal watchdog is disabled# U-Boot enables watchdog (timeout = 5 s); U-Boot main loop will take care of refreshing it# Before giving control to Linux kernel, U-Boot will set up long (e.g 180 seconds) timeout. This is required in order to allow the kernel to complete the boot stage and to run the application that will handle the watchdog refresh# Once the kernel boot process has completed, watchdog application will open the watchdog device file and will take care of its refresh (timeout = 10 s) To enable watchdog support in U-Boot, source code must be modified and the bootloader must be recompiled. Usually, this means enabling CONFIG_WATCHDOG and CONFIG_xxx (where xxx is the name of the watchdog device). Once Linux is started (and if the kernel is compiled with watchdog support), watchdog is refreshed by a simple application like the one shown below: <pre>#include <stdio.h>#include <unistd.h>#include <fcntl.h>#include <stdlib.h>int main(int argc, const char *argv[]){ int fd=open("/dev/watchdog",O_WRONLY); if(fd==-1){ perror("watchdog"); exit(1); } while(1){ write(fd,"\0",1); fsync(fd); sleep(5); }}</pre> This requires the character device file:<pre>crw-r--r-- 1 root root 10, 130 Oct 3 2006 /dev/watchdog</pre> which can be created using the following command:<pre>mknod /dev/watchdog c 10 130</pre>   (*) Enabling the watchdog functionality has several system-level consequences as well. As such, system integrators need the freedom to decide whether to enable it or not. For instance, there are applications for which the system integrators do not want to enable the watchdog. They prefer that their final users are aware of software hangs and that the users reset the board manually. Of course, this approach might not make any sense in case of unattended devices.  == Setting the MAC address of network interfaces== In case the system provides an Ethernet interface, it must be guaranteed that each device is delivered with a unique MAC address. MAC addresses are managed by IEEE Registration Authority: IEEE Registration Authority IEEE Standards Department 445 Hoes Lane Piscataway NJ 08854 Phone: (732) 562-3813 Fax: (732)562-1571 http://standards.ieee.org/contact/form.html For more details see also:<br>* http://standards.ieee.org/develop/regauth/index.html* http://standards.ieee.org/develop/regauth/tut/index.html '''DAVE Embedded Systems''' owns an IAB (Individual Address Block, a set of 4096 addresses), that is in the public listing, so everyone can find out that an address is associated to '''DAVE Embedded Systems'''. Note that the registration authority provides only IABs and OUIs (16000000+ addresses), and that a company is not allowed to request another IAB until at least 95% of the MAC addresses of the previous IAB have been used. Customers who build their products using '''DAVE Embedded Systems'''' SOMs (Naon, Lizard, Qong, Zefeer,...) usually provide MAC numbers by themselves by acquiring them from IEEE. In fact, there are many reasons for this. Three can be stressed: * A CPU module is NOT an end-product. It is not a product that goes directly to the final user as a LAN PCI board, or a printer server. So, in case of CPU modules, who gets a CPU module and build its own product with it, is responsible for handling the MAC address.* Even if '''DAVE Embedded Systems''' programs the MAC address in flash (as an example) at the manufacturing stage, the customer may erase, overwrite, modify this number for the actual CPU module. Also, the strategy and the position (NOR, NAND, E2PROM,...) of the MAC address may vary. '''DAVE Embedded Systems''' cannot guarantee - in other words - that MAC address is maintained in the form and position it had when delivered.* An end-product hosting a '''DAVE Embedded Systems''' CPU module is not always a '''DAVE Embedded Systems'''' product. When it is (and there are some examples), '''DAVE Embedded Systems''' puts the proper MAC address on the product. When it's not, DAVE can't provide MAC addresses: as already stated, the list of DAVE's MAC addresses is public, and by reading this list everybody can see that the product manufacturer is '''DAVE Embedded Systems''', which is not true. ==Handling different product models==It is quite common to manufacture different product models on the top of the same hardware platform. In such cases, it is convenient to use a unified software for all models. DAVE Embedded Systems provides the [[ConfigID_and_UniqueID|ConfigID]] mechanism that can be exploited to achieve this goal. ConfigID can be exploited at boot loader level, at kernel level, and at application level.
4,650
edits

Navigation menu