XELK-AN-008: How to use systemd on an Embedded system

Info Box Applies to Axel Ultra Applies to AXEL ESATTA Applies to Axel Lite Applies to AXEL Lite Application Notes Applies to SMARX Application Notes Applies to SBC AXEL Applies to SBC Axel Application Notes

This application note has been validated starting from the XELK 4.0.0 kit version.

History

Version	Date	XELK version	Notes
1.0.0	Sep 2019	4.0.0

Introduction

Starting from XELK 4.0.0 the root file system generated by NXP Yocto recipes produces a root file system using systemd.

systemd is a System and Service Manager which has enough different settings and configuration from SystemV(SysV) which was used on all XELK BSPs up to XELK 3.0.0.

This application note is not a complete systemd user's guide but collects some useful hints that can be used for getting familiar with systemd. There is a plenty of documentation and User's Guide available for systemd, but some simple examples - that can be found here below - may simplify the systemd approach for beginners.

Architecture

Here below, a picture (from wikipedia) showing the main systemd components:

Configuring systemd

The most used commands on a Linux embedded system are the commands used for: start a service, looking at logging, evalutate the boot time and configuring the network interface.

In the following paragraphs, there are the related commands used for these tasks.

Manage services

systemctl is the main command utility and primary tool for managing the systemd daemons/services such as start, restart, stop, enable, disable, reload & status.

It is possible to display all started services with the following userspace command:

systemctl -t service

It is possible to display all services (including disabled and stopped services) with:

systemctl -t service --all

service commands

Starting a service from userspace:

systemctl start <service_name>

Stopping a service from userspace

systemctl stop <service_name>

Starting a service at boot time:

systemctl enable <service_name>

Disabling service (already started at boot time):

systemctl disable <service_name>

mask a service

There is a third level for stopping a service other than stop and disable: it is the command mask.

It stops the service and it will not possible to start it again using start. Using systemctl it is possbile to mask/unmask a service:

root@imx6qxelk:~# systemctl mask emergency
Created symlink /etc/systemd/system/emergency.service → /dev/null.

If we will try to start it:

root@imx6qxelk:~# systemctl start emergency
Failed to start emergency.service: Unit emergency.service is masked.

In this way, the service will not be used as a dependency in the Unit

The reverse command is unmask:

root@imx6qxelk:~# systemctl unmask emergency
Removed /etc/systemd/system/emergency.service.

Migrating from SystemV to systemd

start

Considering a SystemV script executing the start() function as in the following example:

start() {
  echo "Starting My Custom Service..."
  /usr/bin/myservice -D
}

The related command is executed in the custom service /usr/bin/myservice with the same -D parameter. It is possibile to use the ExecStart=:

[Service]
ExecStart=/usr/bin/myservice -D

restart

The same SystemV script may use special commands for restarting the service like reboot() function:

reboot() {
  echo "Reloading My Custom Service..."
  /usr/bin/myservice reload
}

which is equivalent to use ExecReload=:

[Service]
ExecReload=/usr/bin/myservice reload

stop

The stop() function in the script will become ExecStop=:

SystemV:

stop() {
  echo "Stopping My Custom Service..."
  /usr/bin/myservice shutdown
}

systemd:

[Service]
ExecStop=/usr/bin/myservice shutdown

Configuring the network interfaces

One of the most systemd configuration used is the Network configuration.

Wired interface

systemd uses a slightly different configuration mechanism than SystemV. The configuration file is the following one with an example of configuration:

/etc/systemd/network/20-eth0.network

[Match]
Name=eth0

# Prevent the interface loading if the kernel boots from nfs
KernelCommandLine=!nfsroot

[Network]
Address=192.168.0.120
Gateway=192.168.0.254
DNS=192.168.0.1
#DNS=8.8.8.8

Note:

The DNS is used only if the systemd-resolved service is enabled and the /etc/resolv.conf has a symbolic link to /run/systemd/resolve/stub-resolv.conf

ln -sf /run/systemd/resolve/stub-resolv.conf /etc/resolv.conf

Activate the new network configuration restarting the systemd services:

systemctl restart systemd-networkd
systemctl restart systemd-resolved

Wireless interface

wpa_supplicant

wpa_supplicant provides different services on systemd:

• wpa_supplicant.service uses D-Bus, recommended with the NetworkManager
• wpa_supplicant@interface.service uses the interface name (like wlan0) as parameter and executes the wpa_supplicant daemon on that interface. The configuration file is /etc/wpa_supplicant/wpa_supplicant-interface.conf

For enabling the interface at boot time it is required to enable the service:

systemctl enable wpa_supplicant@interface

wlan configuration example

Assuming wlan0 as the wireless interface name, the configuration file examples are the following one:

/etc/systemd/network/wlan0.network

[Match]
Name=wlan0
[Network]
# Uncomment for DHCP
#DHCP=yes
Address=192.168.1.120
Gateway=192.168.1.254
DNS=8.8.8.8

/etc/wpa_supplicant/wpa_supplicant-wlan0.conf

ctrl_interface=/var/run/wpa_supplicant
eapol_version=1
ap_scan=1
fast_reauth=1

network={
    ssid="SSID1"
    psk="password1"
    priority=1
}
network={
    ssid="SSID2"
    psk="password2"
    priority=2
}

For automatically creating the network configuration, the following command can be used:

wpa_passphrase <ESSID> <passphrase> >> /etc/wpa_supplicant/wpa_supplicant-wlan0.conf

Then, the service should be enabled on the wlan0 interface for let systemd to start it using the (just) created configuration file wpa_supplicant-wlan0.conf:

systemctl enable wpa_supplicant@wlan0

Logging with systemd (journalctl)

systemd has its own logging process called journal avoiding to start the syslog daemon. For the status information it is possible to use journalctl.

journalctl ha many command line switches, to customize its behavior and filter log data (a good reference can be found here

For example, to display the new log messages (similar to tail -f /var/log/messages) add the -f option

With -p it's possible to set the log priority

journalctl -p LEVEL

Where LEVEL can be the number or keyword of the following table (sorted from higher to lower priority). Specifying a priority will display messages marked with that priority or higher

Value	Keyword
0	emerg
1	alert
2	crit
3	err
4	warning
5	notice
6	info
7	debug

User can filter by arbitrary time limits using the --since and --until options, which restrict the entries displayed to those after or before the given time, respectively. E.g.:

root@imx6qxelk:/home# journalctl --since "20 min ago" 
-- Logs begin at Wed 2019-06-26 13:22:41 UTC, end at Mon 2019-07-08 13:22:01 UTC. --
Jul 08 13:11:54 imx6qxelk kernel: ERROR: v4l2 capture: slave not found!
Jul 08 13:11:54 imx6qxelk kernel: ERROR: v4l2 capture: slave not found!
Jul 08 13:11:54 imx6qxelk kernel[551]: [ 3157.796945] ERROR: v4l2 capture: slave not found!
Jul 08 13:11:54 imx6qxelk kernel[551]: [ 3157.801690] ERROR: v4l2 capture: slave not found!
Jul 08 13:11:54 imx6qxelk kernel: ERROR: v4l2 capture: slave not found!
Jul 08 13:11:54 imx6qxelk kernel: ERROR: v4l2 capture: slave not found!
Jul 08 13:11:54 imx6qxelk kernel[551]: [ 3157.859371] ERROR: v4l2 capture: slave not found!
Jul 08 13:11:54 imx6qxelk kernel[551]: [ 3157.864406] ERROR: v4l2 capture: slave not found!

For displaying the log related to a specific Unit, use the -u option, e.g.:

root@imx6qxelk:/home# journalctl -u systemd-networkd  
-- Logs begin at Wed 2019-06-26 13:22:41 UTC, end at Mon 2019-07-08 13:25:01 UTC. --
Jul 05 11:02:13 imx6qxelk systemd-networkd[572]: Enumeration completed
Jul 05 11:02:13 imx6qxelk systemd-networkd[572]: eth0: IPv6 enabled for interface: Success
Jul 05 11:02:15 imx6qxelk systemd-networkd[572]: eth0: Gained carrier
Jul 05 11:02:16 imx6qxelk systemd-networkd[572]: eth0: Gained IPv6LL
Jul 05 11:02:29 imx6qxelk systemd-networkd[572]: eth0: Configured

For displaying the log related to a specific /dev device, just add it to the command line:

root@imx6qxelk:~# journalctl /dev/fb0
-- Logs begin at Wed 2019-06-26 13:22:41 UTC, end at Thu 2019-07-11 09:07:01 UTC. --
Jun 26 13:22:41 imx6qxelk kernel: mxc_sdc_fb fb@0: registered mxc display driver ldb
Jun 26 13:22:41 imx6qxelk kernel: mxc_sdc_fb fb@0: using reserved memory region at 0x8e000000, size 2 MiB
Jun 26 13:22:41 imx6qxelk kernel: mxc_sdc_fb fb@0: assigned reserved memory node splashscreen
Jun 26 13:22:41 imx6qxelk kernel: mxc_sdc_fb fb@0: using memory region 0x8e000000 0x8e1fffff

For displaying the log related to a user ID, use _UID= parameter

root@imx6qxelk:~# id messagebus
uid=995(messagebus) gid=993(messagebus) groups=993(messagebus)
root@imx6qxelk:~# journalctl _UID=993
-- Logs begin at Wed 2019-06-26 13:22:41 UTC, end at Thu 2019-07-11 09:14:01 UTC. --
Jul 10 14:42:48 imx6qxelk systemd-timesyncd[423]: Network configuration changed, trying to establish connection.
Jul 10 14:43:02 imx6qxelk systemd-timesyncd[423]: Network configuration changed, trying to establish connection.
Jul 11 07:38:31 imx6qxelk systemd-timesyncd[423]: Synchronized to time server 216.239.35.8:123 (time3.google.com).

Analyze the boot time

Boot time analysis is one of the most important and interesting activity for an embedded system: systemd provide an userspace command called systemd-analyze to help in this (hard) task

The systemd-analyze command list how many services are running on the system and how long they took for starting at the last boot.

systemd-analyze provides a good level of boot time information for further optimizations:

time

The time parameter gives the total amount of seconds used for starting the kernel and reaching the userspace.

root@imx6qxelk:~# systemd-analyze time 
Startup finished in 5.109s (kernel) + 4.771s (userspace) = 9.880s

blame

The blame parameter gives the list of started services and how long they took for starting:

root@imx6qxelk:~# systemd-analyze blame 
          3.608s dev-mmcblk0p2.device
           547ms systemd-remount-fs.service
           545ms systemd-vconsole-setup.service
           544ms kmod-static-nodes.service
           503ms systemd-udev-trigger.service
           426ms systemd-journal-flush.service
           407ms tmp.mount
           371ms systemd-logind.service
           327ms systemd-journald.service
           317ms systemd-networkd.service
           275ms systemd-timesyncd.service
           257ms systemd-sysctl.service
           204ms ofono.service
           203ms systemd-modules-load.service
           194ms sys-kernel-config.mount
           188ms sys-kernel-debug.mount
           177ms sshd.socket
           161ms psplash-start.service
           138ms systemd-random-seed.service
           138ms sys-fs-fuse-connections.mount
           129ms systemd-udevd.service
           129ms systemd-update-utmp.service
           128ms systemd-tmpfiles-setup-dev.service
           124ms rc-local.service
            98ms systemd-tmpfiles-setup.service
            91ms psplash-quit.service
            90ms systemd-resolved.service
            89ms systemd-backlight@backlight:backlight.service
            63ms dev-mmcblk0p1.device
            41ms var-volatile.mount
            33ms systemd-update-utmp-runlevel.service

critical-chain

The critical-chain parameter shows the startup process flow and the time consumed by each service.

Here below a picture showing an example of critical path:

Services and targets

systemd manages not only services but many different objects called Unit. Unit are related to the resources that systemd can manage. Unit configurations are defined into the Unit files.

Units categoris (identified by the file extension) are:

   .service
   .target
   .socket
   .device
   .mount
   .automount
   .swap
   .path
   .timer
   .snapshot
   .slice
   .scope

Major interesting Units are services and targets. They will be analyzed in the following paragraphs.

Targets

Targets are used by systemd for having a synchronization mechanism between different services at boot time or during run-time changes.

They can be used for set the system to a new state.

All services linked to a target are linked to the modification to the same target. These can be seen in a similar way of SystemV runlevels with many other added functionalities.

Target and runlevels

Here below there is a list of power on/off targets and related SystemV runlevels:

Description	SystemV (runlevel)	systemd (target)
System halt	0	runlevel0.target, poweroff.target
Single user mode	1, s, single	runlevel1.target, rescue.target
Multi user	2	runlevel2.target, multi-user.target
Multi user with network	3	runlevel3.target, multi-user.target
Experimental	4	runlevel4.target, multi-user.target
Multi user with network, graphical mode	5	runlevel5.target, graphical.target
Reboot	6	runlevel6.target, reboot.target

multi-user target can be identified as the runlevel 3.

Into the following directory:

/etc/systemd/system/<target_name>.target.wants

there is a list of services related to that target.

For example:

root@imx6qxelk:~# ls /etc/systemd/system/multi-user.target.wants/
atd.service	       busybox-syslog.service  gpuconfig.service  ofono.service		systemd-networkd.service
avahi-daemon.service   connman.service	       mytest.service	  psplash-quit.service	systemd-resolved.service
busybox-klogd.service  crond.service	       ntpdate.service	  remote-fs.target

Active targets

It is possible to display all active targets with:

systemctl -t target

Changing a target

systemctl isolate graphical

The actual target is shown with:

systemctl get-default

Changing the default target:

systemctl set-default multi-user

Unit files

For a complete information on Unit please look to the documentation page

Here below you can find an extract for the main used topics and configuration descriptions.

Location Path

Units are configured by systemd using configuration files that can be found in different directories. Each of them has different priority and behaviour:

Path	Description
/lib/systemd/system	This directory stores a copy of configuration files: this is the default destination for new installed configuration file. Typically files in this directory should not be modified by the user.
/etc/systemd/system	This is the directory where to store a new Unit or to modify an existing one. The files present in this directory have the highest priority.
/run/systemd/system	The files present in this directory have higher priority only respect the ones on /lib/systemd/system. Systemd creates these configuration files dinamically at runtime; modification on this directory can be used for testing a runtime behaviour for a Unit but all modifications will be lost at next boot.

[Unit] section options

This section is used for defining the metadata and relations between different Unit

Please find below the main properties description:

Property	Function
Description=:	Name and function
Documentation=:	URI for the documentationvv
Requires=:	List of Units dependencies. For successfully executing this Unit, all listed dependency should be activated without errors, otherwise this Unit return fail.
Wants=:	Similar to a Requires but weaker. If the Unit listed are not found or return fail, this Unit are executed anyway. this is the recommended method to be used.
BindsTo=:	Similar to Requires but it does a Stop for the Unit when the listed Unit are terminated.
Before=:	The Unit listed will not be executed until this Unit will not change to started. This is used for an order of Units executions.
After=:	The Unit listed will be started before this Unit. This is used for an order of Units executions.
Conflicts=:	The Unit listed cannot be executed simultaneously to this Unit.

[Install] section options

This section is optional but is commonly used for defining a Unit behaviour when it will be executed during enable or disable commands.

Property	Function
WantedBy=:	This is similar to the Wants= on [Unit] section but allows to mantain the top Unit more clean. When the Unit will be enabled, a directory on /etc/systemd/system will be created with the Unit name adding .wants to the name. Inside this directoty a symbolic link to the Unit is created. Example: current Unit has WantedBy=multi-user.target a directory /etc/systemd/system/multi-user.target.wants will be created the symbolic link to the Unit will be created inside the new directory disabling the Unit the symbolic link is deleted and the relation is then removed.
RequiredBy=:	This is similar to WantedBy= but a dependency cause a fail if not satisfied. When the Unit is enabled, a directory with added .requires will be created
Also=:	When the Unit is enabled, also the listed Units are enabled too.

Specific sections

Some Unit have specific sections based on their characteristic. The most important is the section Service related to the Unit .service

Please find more information at the documentation page

[Service] section

Used for providing configurations for the services.

Type

Type= uses one of the (main) following values:

Value	Description
simple	Default configuration for a service when specified ExecStarts=
forking	the process will call a fork() when starts causing the father to exit. This informs systemd that the process is still alive even if the father has been terminated.
oneshot:	the process has a very short execution time and then systemd should wait for its termination before continuing with other Units. this is the default configuration if ExecStarts= is not specified.
dbus	the Unit will acquire the name on the D-Bus. systemd will continue to process the other Units
notify	the service will notify when completely initialized. systemd will wait for the notification before continuing with the following Units
idle	the service will not be executed until all active jobs are dispatched.

Other options

Value	Description
ExecStarts=:	Specifiy the full path and parameters for executing a service. If preceded by a "-" this inform that the command failure can be accepted.
ExecStartsPre=:	used for adding more commands to be executed before starting the main process. May be used multiple times specifying the complete path and command parameters.
ExecReload=:	commands to be executed for reloading the service configuration.
ExecStop=:	commands required for stopping the service. If missing, the service will be killed.
ExecStopPost=:	commands to be executed after the service has been stopped..
RestartSec=:	time to sleep (seconds) before restarting the service.
Restart=:	restart conditions for systemd to be checked before restarting the service (if terminated). Can be set to "always","on-success", "on-failure", "on-abnormal", "on-abort", or "on-watchdog".
TimeoutSec=:	time to sleep during start or stop before considering the process failed on start or stop. Start and stop timeout can be set with different values using TimeoutStartSec= and TimeoutStopSec=

Putting it all together: create a new service

For creating a new service the following file has to be created:

/etc/systemd/system/<service_name>.service

Service example

The following paragraph shows how to create a new service called iperf3 executing the iperf3 command in server mode

/etc/systemd/system/iperf3.service

[Unit]
Description=iperf3 server mode
After=network.target
StartLimitIntervalSec=0

[Service]
Type=simple
Restart=0
RestartSec=1
User=root
ExecStart=/usr/bin/iperf3 -s

[Install]
WantedBy=multi-user.target

Basic settings

Parameter	Description
After	The executed command (iperf3) requires the network interface to be already active, so we use After for this purpose.
Restart	This is configured with 0 for disabling the service after it has been run.
RestartSec	time sleep before restarting the service; default value is 100ms.
User	configures the user or group used for executing the service.
ExecStart	command to be executed when the service will be started (in our case iperf3).
WantedBy	defines which target is used related to the service started.

Running a service

Starting the service from userspace:

systemctl start iperf3

Starting the service at boot time:

systemctl enable iperf3