Changes

Deployment of [[Embedded Linux]] systems is the typical operation which follows the development phase. When the application is ready and fully tested in the develoment development environment, it's time to take the system to the field for the “real work”. This phase brings a lot of concerns to cope with, for example creating a suitable root filesystem, saving the data properly, implement and implementing successful on-the-field update strategies. This how-to guide explains how to solve lists some of the problems connected common issues related to the deployment of an embedded linux Linux system.

The host (usually a PC or a virtual machine running the Linux operating system) is used by the developer to (cross-)compile the code that will run on the target, for example a Dave '''DAVE Embedded Systems''' ARM CPU module such as Lizard or Naon. The Linux kernel running on the target is able to mount the root file system from different physical media. During the software development, it is very common to use a directory exported via [[W:Network File System|NFS]] by the host for this purpose. Moreover, the linux kernel is usually retrieved by a simple network transfer protocol like [http://en.wikipedia.org/wiki/Trivial_File_Transfer_Protocol tftp].

When the system is ready to move to the field, most of the times the particular pysical link between host and target used at the development stage must be removed. In the worst case, the system must run without any NFS filesystem or file transfer services, relying only on its hardware resources (for example, the on-board ram RAM and flash memories). Resources that These resources are obviously limited, due to the nature of the embedded systems. Generally speaking, the procedure used to deploy the system configuration highly depends on the specific application; however, some topics are quite common. The following sections shine a light on these topics. ==Using an evaluation kit for the development stage==Often, it is convenient to carry out the development stage with an evaluation kit such as [[AXEL_ULite_and_SBC_Lynx_Embedded_Linux_Kit_(XUELK)|XUELK]] and [[Axel_Embedded_Linux_Kit_(XELK)|XELK]]. This is the case, for instance, when the final hardware platform is not available yet. It is worth remembering that, in this scenario, '''it is mandatory to port the software Board Support Package (BSP) to the final hardware platform when migrating from the evaluation kit'''. Conventionally, the BSP consists of:* Firmware (if any)* Bootloader (typically U-Boot) and, optionally, its device tree* Linux kernel and its device tree* Root file system.'''If not properly ported, the BSP may cause irreversible damage to the platform or severe malfunctioning as it is strictly related to the underlying hardware.''' Therefore, '''it is strongly discouraged to run the binary files released for DAVE Embedded Systems' evaluation kits on different platforms unless you know what you are doing.'''

Recent versions At the time of this writing (May 2019), '''Yocto has become by far the [http://wwwmost popular build system in the Linux embedded arena'''.denxLarge silicon vendors such as NXP, Texas Instruments, and Xilinx base their BSPs on Yocto.de/wiki/ELDK-5 That's why '''all the Linux development kits released by DAVE Embedded Linux Develpment Kit from Denx] Systems are in turn based on Yoctofor some years now'''.

=== Adding libraries Application dependancies ===

=Non-volatile memories partitioning scheme = Deploying In general, many different schemes can be implemented. Cost, reliability, speed are just some of the root file system == = Watchdog =parameters to be considered when designing the partitioning scheme. [[Standalone_boot_(BELK/BXELK)|This article]] shows two quite different schemes on the same platform, for example.

== The serial port ==U-Boot implements a text console on the serial port. This console can be used to stop Configuring the startup sequence to allow an interactive session with the human operator. This is very useful during debug operations, but by default any character the bootloader receives during the startup sequence stops this process. As a side effect, any spurious character received from the serial port devoted to the console is able to prevent the bootloader to complete the automatic boot process (and typically to start the operating system).  loader== Autoboot configuration ==For this reason before moving to the field it is highly recommended to configure the bootloader to halt the sequence when receiving a specific string. Fortunately, Autoboot process is deeply configurable: parameters defining the retry behaviour and the strings used to stop booting can be specified. Please read the README.autoboot file, provided inside the documentation directory of the U-Boot sources, for more details. = Setting the MAC address = In case the system provides Ethernet interface, it must be guaranteed that each device is delivered with a unique MAC address. MAC addresses are managed by IEEE Registration Authority: IEEE Registration Authority IEEE Standards Department

Piscataway NJ 08854 PhoneFor this reason, before moving to the field it is highly recommended to configure the bootloader to halt the sequence when receiving a specific string instead. Fortunately, Autoboot process is deeply configurable: (732) 562parameters defining the retry behaviour and the strings used to stop booting can be specified. Please read the README.autoboot file, provided inside the documentation directory of the U-3813Boot sources, for more details.

Fax: (732) 562===Automating operations at boot loader level===Modern boot loaders such as U-1571 http://standardsBoot allows automating boot operations in an extremely flexible fashion.ieeeThanks to the scripting capabilities, it is possible to execute complex scripts.org/contact/form.html For more details , see also:<br>* httpfor instance [https://standards.ieeeelinux.org/develop/regauth/indexECE597_boot.html* httpscr this tutorial] or [https://standards.ieeeelinux.org/develop/regauth/tut/index.html Dave owns an IAB (Individual Address Block, a set of 4096 addresses), that is in the public listing, so everyone can find out that an address is associated to Dave. Note that the registration authority provides only IABs and OUIs (16000000+ addresses), and that a company is not allowed to request another IAB until at least 95% of the MAC addresses of the previous IAB have been used. Customers who build their products around DAVE's processor modules (Naon, Lizard, Qong, Zefeer,...) usually provide MAC numbers by themselves by acquiring them from IEEE. In fact there are many reasons for that. Three can be stressed: * A CPU module is NOT an end-product. It is not a product that goes directly to the final user as a LAN PCI board, or a printer server. So, in case of CPU modules, who gets a CPU module and build its own product with it, is responsible for handling the MAC address.* Even if DAVE programs the MAC address in flash (as an example) at manufacturing stage, customer may erase, overwrite, modify this number for the actual CPU module. Also, the strategy and the position (NOR, NAND, E2PROM,...) of the MAC address may varyECE597_boot. DAVE cannot guarantee - in other words - that MAC address is maintained in the form and position it had when delivered.* An end-product hosting a DAVE's CPU module is not always a DAVE's product. When it is (and there are some examples), DAVE puts the proper MAC address on the product. When it's not, DAVE can't provide MAC addresses: as already stated, the list of DAVE's MAC addresses is public, and reading scr this list someone can see that the product manufacturer is DAVE, which is not truepage].

Basically , we can upgrade the system through the bootloader or through the kernel.

When the system doesn't allow a remote access, an operator must locally access the system through its user interface (in the best case) or through the serial port (in the worst case). If the

More information are is available in README, INSTALL and MULTI files included in the Dropbear distribution. Please note that for recent systems, as Lizard and Naon, Dropbear can be installed from '''pre-built packages''' (please refer to the Angstrom distribution's package-manager documentation). =Security=Nowadays, a large number of newly designed embedded systems are used in Internet-connected products. As such, these systems have to face severe issues in terms of cybersecurity. This topic is huge and it is obviously beyond the scope of this document. It is worth remembering, however, that modern SOCs provide a rich set of native functionalities that address this issue. For instance, see [[XUELK-WP-001:_Secure_boot_on_iMX6UL|this page]] or [[BELK-TN-001:_Real-timeness,_system_integrity_and_TrustZone%C2%AE_technology_on_AMP_configuration|this technical note]]. These documents show how some of such functionalities were leveraged on DAVE Embedded Systems products. For a more general discussion about the security of Internet-connected devices, see for example the [https://www.iotsecurityfoundation.org/ IoT Security foundation website] or [https://www.microsoft.com/en-us/research/publication/seven-properties-highly-secure-devices/ this paper] by Microsoft. =Licensing=It is known that a GNU/Linux system is based on a large amount of open source software. Consequently, the manufacturers of a product embedding such software must deal with the open source license compliance. This topic is beyond the scope of this document, but a couple of links are provided anyway:*[https://elinux.org/Legal_Issues This article] provides an overview of license-related legal issues*Yocto build system generates automatically a manifest file to make the compliance management easier. For more details, please refer to [[https://www.yoctoproject.org/docs/1.8/dev-manual/dev-manual.html#maintaining-open-source-license-compliance-during-your-products-lifecycle|this link]] =Certifications=In general, all the products must conform to a set of standards in order to get the certificates required to be sold ([https://en.wikipedia.org/wiki/CE_marking CE marking] and [https://en.wikipedia.org/wiki/FCC_Declaration_of_Conformity FCC_Declaration_of_Conformity] are two well-known examples. In many cases, software modifications to the Board Support Package (BSP) or the user applications are required to pass the compliance tests dictated by these regulations (*). In a typical scenario, these modifications are usually carried out after the development of the user applications. Nevertheless, these changes must be committed to the source repositories in order to be part of the production images that are stored into the product.   (*) A common example is the enabling of the spread-spectrum technique to reduce the electromagnetic emission peak associated with a digital clock, which is usually implemented at boot loader level. =Misc=== Watchdog == Typically, during application development, the watchdog device included in the embedded system is turned off. Before moving to the field, enabling the watchdog is often a mandatory choice (*). The use of this peripheral is a little bit tricky because it involves both U-Boot and Linux. The following sequence shows the typical scenario when the system is working on the field:  # Processor comes out of reset; internal watchdog is disabled# U-Boot enables watchdog (timeout = 5 s); U-Boot main loop will take care of refreshing it# Before giving control to Linux kernel, U-Boot will set up long (e.g 180 seconds) timeout. This is required in order to allow the kernel to complete the boot stage and to run the application that will handle the watchdog refresh# Once the kernel boot process has completed, watchdog application will open the watchdog device file and will take care of its refresh (timeout = 10 s) To enable watchdog support in U-Boot, source code must be modified and the bootloader must be recompiled. Usually, this means enabling CONFIG_WATCHDOG and CONFIG_xxx (where xxx is the name of the watchdog device). Once Linux is started (and if the kernel is compiled with watchdog support), watchdog is refreshed by a simple application like the one shown below: <pre>#include <stdio.h>#include <unistd.h>#include <fcntl.h>#include <stdlib.h>int main(int argc, const char *argv[]){ int fd=open("/dev/watchdog",O_WRONLY); if(fd==-1){ perror("watchdog"); exit(1); } while(1){ write(fd,"\0",1); fsync(fd); sleep(5); }}</pre> This requires the character device file:<pre>crw-r--r-- 1 root root 10, 130 Oct 3 2006 /dev/watchdog</pre> which can be created using the following command:<pre>mknod /dev/watchdog c 10 130</pre>   (*) Enabling the watchdog functionality has several system-level consequences as well. As such, system integrators need the freedom to decide whether to enable it or not. For instance, there are applications for which the system integrators do not want to enable the watchdog. They prefer that their final users are aware of software hangs and that the users reset the board manually. Of course, this approach might not make any sense in case of unattended devices.  == Setting the MAC address of network interfaces== In case the system provides an Ethernet interface, it must be guaranteed that each device is delivered with a unique MAC address. MAC addresses are managed by IEEE Registration Authority: IEEE Registration Authority IEEE Standards Department 445 Hoes Lane Piscataway NJ 08854 Phone: (732)562-3813 Fax: (732) 562-1571 http://standards.ieee.org/contact/form.html For more details see also:<br>* http://standards.ieee.org/develop/regauth/index.html* http://standards.ieee.org/develop/regauth/tut/index.html '''DAVE Embedded Systems''' owns an IAB (Individual Address Block, a set of 4096 addresses), that is in the public listing, so everyone can find out that an address is associated to '''DAVE Embedded Systems'''. Note that the registration authority provides only IABs and OUIs (16000000+ addresses), and that a company is not allowed to request another IAB until at least 95% of the MAC addresses of the previous IAB have been used. Customers who build their products using '''DAVE Embedded Systems'''' SOMs (AXEL Lite, AXEL ULite, BORA, ORCA,...) usually provide MAC numbers by themselves by acquiring them from IEEE. In fact, there are many reasons for this. Three can be stressed: * A CPU module is NOT an end-product. It is not a product that goes directly to the final user as a LAN PCI board, or a printer server. So, in case of CPU modules, who gets a CPU module and build its own product with it, is responsible for handling the MAC address.* Even if '''DAVE Embedded Systems''' programs the MAC address in flash (as an example) at the manufacturing stage, the customer may erase, overwrite, modify this number for the actual CPU module. Also, the strategy and the position (NOR, NAND, E2PROM,...) of the MAC address may vary. '''DAVE Embedded Systems''' cannot guarantee - in other words - that MAC address is maintained in the form and position it had when delivered.* An end-product hosting a '''DAVE Embedded Systems''' CPU module is not always a '''DAVE Embedded Systems'''' product. When it is (and there are some examples), '''DAVE Embedded Systems''' puts the proper MAC address on the product. When it's not, DAVE can't provide MAC addresses: as already stated, the list of DAVE's MAC addresses is public, and by reading this list everybody can see that the product manufacturer is '''DAVE Embedded Systems''', which is not true. ==Handling different product models==It is quite common to manufacture different product models on the top of the same hardware platform. In such cases, it is convenient to use a unified software for all models. DAVE Embedded Systems provides the [[ConfigID_and_UniqueID|ConfigID]] mechanism that can be exploited to achieve this goal. ConfigID can be exploited at boot loader level, at kernel level, and at application level.  = Deployment Checklist =Here below, a list of topics that should be taken into account for Product design and deployment on the field.  This checklist tries to cover the most known project/product aspects which should be - ''at least'' - carefully understood and answered before a product is installed on the field. == Non volatile storage ==* Partitioning scheme** Endurance/reliability: NOR/NAND/SD/eMMC evaluation respect to reliability, costs, write cycles, data retention, data dimension** Redundant scheme or Recovery partition (A/B or A + recovery)== Boot sequence ==* boot time requirements (in terms of ''seconds'') and behaviours* BSP boot time analisys** u-boot, kernel, minimal rfs ** application startup== Applications ==* special framework required: Java, Qt, Javscript engines== Accessibility ==*serial console: availability (vs security)** startup configuration: boot stop with special character== Upgrade ==* upgrade on the field* Local or Remote** with recovery or redundant partitions*** tracking of upgrade failures for maintenance information** On The Field: removable peripherals usage (SD, USB) and ''"non expert"'' upgrade== Security ==* "Security by design": take into account security requirements at the beginning of the project* useful guideline formalized by da Microsoft for [https://azure.microsoft.com/it-it/blog/azure-sphere-is-here-public-preview-and-dev-kits-now-available Azure Sphere]== Recovery and service ==* complete automatic Disaster recovery* complete - manually assisted - Disaster recovery* partial Disaster recovery* restore to Manufacturer configuration* Service Mode== Licensing==* specific license requirements, i.e. Qt library== Certification ==* EMC certification* other specific certification requests (or document requirement like MTBF, etc.)== Various ==* Watchdog** in case of WDT intervention: tracking of watchdog event for software analysis* MAC addresses handling: see [http://standards.ieee.org/develop/regauth/index.html IEEE registration authority]* runtime automatic detection and configuration: possible usage of [[ConfigID_and_UniqueID | ConfigID]] technique