Deploying Embedded Linux Systems

Info Box Applies to Linux

Introduction

Deployment of Embedded Linux systems is the typical operation which follows the development phase. When the application is ready and fully tested in the develoment environment, it's time to take the system to the field for the “real work”. This phase brings a lot of concerns to cope with, for example creating a suitable root filesystem, saving the data properly, implement successful on-the-field update strategies. This how-to guide explains how to solve the problems connected to the deployment of an embedded linux system.

The development environment

The following figure illustrates the typical developing environment for an Embedded Linux system: it is composed by a host machine and a target machine.

The host (usually a PC or a virtual machine running the Linux operating system) is used by the developer to (cross-)compile the code that will run on the target, for example a Dave ARM CPU module such as Lizard or Naon. The Linux kernel running on the target is able to mount the root file system from different physical media. During the software development, it is very common to use a directory exported via NFS by the host for this purpose. Moreover, the linux kernel is usually retrieved by a simple network transfer protocol like tftp.

Moving to the field

When the system is ready to move to the field, most of the times the particular link between host and target must be removed. In the worst case, the system must run without any NFS filesystem or file transfer services, relying only on its hardware resources (for example, the on-board ram and flash memories). Resources that are obviously limited, due to the nature of the embedded systems. Generally speaking, the procedure used to deploy the system configuration highly depends on the specific application; however, some topics are quite common. The following sections shine a light on these topics.

Root file systems

Linux needs a root file system: a root file system must contain everything needed to support the Linux system (applications, settings, data, ..). The root file system is the file system that is contained on the same partition on which the root directory is located. The Linux kernel, at the end of its startup stage, mounts the root file system on the configured root device and finally launches the /sbin/init, the first user space process and "father" of all the other processes. An example of root file system is shown below:

drwxr-xr-x  2 root     root     4096 2011-05-03 11:23 bin/
drwxr-xr-x  2 root     root     4096 2011-04-01 17:20 boot/
drwxr-xr-x  3 root     root     4096 2011-07-07 12:17 dev/
drwxr-xr-x 44 root     root     4096 2011-05-03 19:02 etc/
drwxr-xr-x  4 root     root     4096 2011-04-01 17:35 home/
drwxr-xr-x  5 root     root     4096 2011-05-03 11:23 lib/
drwxr-xr-x 12 root     root     4096 2011-07-07 12:03 media/
drwxr-xr-x  6 root     root     4096 2011-05-19 16:39 mnt/
drwxr-xr-x  2 root     root     4096 2011-03-11 05:21 proc/
drwxr-xr-x  2 root     root     4096 2011-05-03 11:23 sbin/
drwxr-xr-x  2 root     root     4096 2011-03-11 05:21 sys/
lrwxrwxrwx  1 root     root        8 2011-05-30 12:19 tmp -> /var/tmp/
drwxr-xr-x 11 root     root     4096 2010-11-05 19:36 usr/
drwxr-xr-x  8 root     root     4096 2010-12-12 11:30 var/

For more information on the Linux filesystem, please refer to The Linux filesystem explained

Strategies

The integrity of the root file system is mandatory to allow the kernel to complete the boot process and usually it is not required that the whole file system is writeable. For these reasons, usually the file system is splitted in (at least) two parts as shown in the following table:

Part	File System type	Access	Physical medium
Minimal root file system	ext2, cramfs, ..	write protected	ramdisk (*)
Storage file system	UBIFS, JFFS2, YAFFS2, ext2/3, ..	read/write	NOR and NAND flashes, SSD, hard disk, ..

(*) As this file system is mounted over a volatile memory, modifications will be lost when the system will be turned off.

The first part is the actual root file system, it contains the minimum components to allow the system to boot properly and usually it does not require on-the-field upgrading. The other part is used to store applications binaries and files created and/or modified by the user, thus it must be mounted over a non-volatile memory device.

Creating the root file system

Building a root file system from scratch is definitively a complex task because several well known directories must be created and populated with a lot of files that must follow some standard rules. Usually, it's a good idea to start with a pre-packaged root file system, in order to skip the actual creation step, and letting you to work on the customization of the file system. You have two options:

1. start from a big file system and remove all the components (packages, libraries, application binaries, ..) that you don't need
2. start from a small file system and add all the components (packages, libraries, application binaries, ..) that you need

Please see the Embedded distros article for an introduction on Embedded Linux distributions.

If you prefer to build the entire root file system, there are several possibilities that are described in the following sections.

OpenEmbedded

OpenEmbedded is a build framework for Embedded Linux. It offers a cross-compile environment which allows developers to create a complete Linux Distribution for embedded systems. Some of the OpenEmbedded advantages include:

• support for many hardware architectures
• multiple releases for those architectures
• tools for speeding up the process of recreating the base after changes have been made
• easy to customize
• runs on any Linux distribution
• cross-compiles 1000's of packages including GTK+, Qt, the X Windows system, Mono, Java, ...

OpenEmbedded is at the basis of some known distribution, like Angstrom, OpenMoko and Poky and it can target a lot of different targets and architectures. Primarily, the project maintains and develops a collection of BitBake (a task execution manager derived from Gentoo's Portage) recipes The recipes consist of the source URL of the package, dependencies and compile or install options. During the build process they are used to track dependencies, cross-compile the package and pack it up, suitable to be installed on the target device. It's also possible to create complete images, consisting of root file system and kernel. As a first step the framework will build a cross-compiler toolchain for the target platform, then the build system builds all the packages included in the selected BitBake recipe, which can range from a single application to an entire Linux distribution.

Arago

Arago Project targets the TI OMAP, Sitara and DaVinci platform, providing a verified, tested and supported subset of packages and has been created to simplify the standard OpenEmbedded approach (mainly setup and interaction). In fact, setting up a complete OE/Bitbake system is a task recommended only to experienced users/developers, so the availability of a SDK that allows building applications for the target without learning OE/Bitbake is very important for the less-experienced audience.

Customizing the root file system

This step is clearly required to add to the basic root file system your custom application files (libraries, binaries, configuration files, ...)

Adding libraries

The applications executables that you have developed might depend on libraries that are not provided by the basic root file system. In this case these libraries must be added. To find out which libraries your applications depends on you can use the ldd and readelf tools. Please note that often you'll need to rebuild some libraries, cross-compiling them to match your target architecture.

Specific devices

Usually the embedded system provides custom devices for which developers had to write specific device drivers. To enable the support for these devices, the proper device files must be created in /dev.

Drivers built as modules

In embedded system the device drivers are typically statically linked to the kernel. In case they are built as modules you have to:

• install them in the root file system
• provide the command line utilities (*) required to handle the modules

Since the loadable kernel modules is a huge topic, it is recommended to read the Linux Loadable Kernel Module HOWTO available at this URL: http://www.tldp.org/HOWTO/Module-HOWTO/.

(*) for kernels 2.4 you must use the modutils (http://www.kernel.org/pub/linux/utils/kernel/modutils/)
(*) for kernels 2.6 you must use the mod-init-tools (http://www.kernel.org/pub/linux/utils/kernel/module-init-tools/)

Boot scripts

After the kernel is booted and initialized, the kernel starts init, the first user-space application, which commonly is /sbin/init. Init is responsible for starting system processes as defined in the /etc/inittab file. Init typically will start at least an instances of "getty" which waits for console logins which spawn one's user shell process. Upon shutdown, init controls the sequence and processes for shutdown. The init process is never shut down. It is a user process and not a kernel system process although it does run as root.

The boot scripts (typically /etc/inittab and /etc/rc.sh) must be modified in order to automatically execute some operations at boot such as launching user-space applications and mounting file systems (i.e. Sysfs).

Setting the MAC address

In case the system provides Ethernet interface, it must be guaranteed that each device is delivered with a unique MAC address. MAC addresses are managed by IEEE Registration Authority:

IEEE Registration Authority

IEEE Standards Department

445 Hoes Lane

Piscataway NJ 08854

Phone: (732) 562-3813

Fax: (732) 562-1571

http://standards.ieee.org/contact/form.html

For more details see also:

• http://standards.ieee.org/develop/regauth/index.html
• http://standards.ieee.org/develop/regauth/tut/index.html

Dave owns an IAB (Individual Address Block, a set of 4096 addresses), that is in the public listing, so everyone can find out that an address is associated to Dave. Note that the registration authority provides only IABs and OUIs (16000000+ addresses), and that a company is not allowed to request another IAB until at least 95% of the MAC addresses of the previous IAB have been used.

Customers who build their products around DAVE's processor modules (Naon, Lizard, Qong, Zefeer,...) usually provide MAC numbers by themselves by acquiring them from IEEE. In fact there are many reasons for that. Three can be stressed:

• A CPU module is NOT an end-product. It is not a product that goes directly to the final user as a LAN PCI board, or a printer server. So, in case of CPU modules, who gets a CPU module and build its own product with it, is responsible for handling the MAC address.
• Even if DAVE programs the MAC address in flash (as an example) at manufacturing stage, customer may erase, overwrite, modify this number for the actual CPU module. Also, the strategy and the position (NOR, NAND, E2PROM,...) of the MAC address may vary. DAVE cannot guarantee - in other words - that MAC address is maintained in the form and position it had when delivered.
• An end-product hosting a DAVE's CPU module is not always a DAVE's product. When it is (and there are some examples), DAVE puts the proper MAC address on the product. When it's not, DAVE can't provide MAC addresses: as already stated, the list of DAVE's MAC addresses is public, and reading this list someone can see that the product manufacturer is DAVE, which is not true.