XELK-AN-001: Asymmetric Multiprocessing (AMP) on Axel – Linux + FreeRTOS

Info Box Applies to Axel Ultra Applies to Axel Lite Applies to AXEL Lite Application Notes Applies to SBC AXEL

This application note was validated against specific versions of the kit only. It may not work with other versions. Supported versions are listed in the History section.

History

Version	Date	XELK version	Notes
0.9.2	Feb 2015	2.0.0	Update for XELK 2.0.0 release
0.9.3	April 2015	2.0.0	Minor fixes
1.0.0	April 2018	2.1.0	Update for amp-feat-2.1.0-amp branch
1.0.1	October 2018	2.1.0	Minor improvements
1.0.2	October 2018	2.1.0	Added some details about latencystat test application

Introduction

This application note describe how to build the software components required to set up asymmetric multi-processing (AMP for short).

The following configuration required to run Linux OS on first Cortex®-A9 core and FreeRTOS on second Cortex®-A9 core of the Freescale i.MX6 SOC.

• N.B. For the sake of simplicity, in this application note, we are going to use two of the four core inside the iMX6Q cpu. Other configuration are possible but are not relevant for this demo.

The latencystat demo is a RPMsg-based application that exploits sophisticated techniques to handle inter-processors communication and synchronization.

Asymmetric Multiprocessing

Thanks to latest technological improvements, multicore processors are becoming popular in embedded world too. These architectures allows the implementation of processing schemes that were not feasible with traditional single-core CPUs. Among these, one of the most interesting is asymmetric multiprocessing (AMP for short). This configuration permits to address several requirements that the embedded system developers struggle to handle in case of single-core systems.

This application note describes in detail the implementation of Linux/FreeRTOS asymmetric multiprocessing configuration on DAVE Embedded Systems AXEL LITE SoM. This configuration is a typical example about how to leverage AMP flexibility to combine, on one single piece of silicon, the versatility of Linux o.s. for general purpose computation, connectivity and HMI and the determinism of an RTOS to satisfy real-time constraints. Since AXEL family products are all based on Freescale i.MX6 processors, what is described here applies to AXEL ULTRA too.

As known, AMP allows a multicore system to run simultaneously [1] multiple Operating Systems (OS) that are independent of each other. In other words, each core has its own private memory space, which contains the OS and the applications that are to run on that core. In addition, there can be some shared memory space that is used for inter-core communication. This is in contrast to Symmetric Multiprocessing (SMP), in which one OS runs on multiple cores using a public shared memory space.

Thanks to AMP, developers can use open-source Linux and FreeRTOS operating systems and the RPMsg Inter Processor Communication (IPC) framework to quickly implement applications that need to deliver deterministic, real-time responsiveness for markets such as automotive, industrial and others with similar requirements, while preserving the openness of Linux.

The following picture depicts the structure of the system.

AMP configuration

Core #0:

• takes care of boot process
• once Linux gets control of the processor, initializes core #1 in SMP mode
• stops core #1 and switches it to AMP mode
• loads binary image of FreeRTOS that is then executed by core #1

[1] This is true simultaneity because multiple instructions are executed at the same time.

Inter-core communication

Inter-core communication is based on RPMsg framework (please refer to http://omappedia.org/wiki/Category:RPMsg and https://www.kernel.org/doc/Documentation/rpmsg.txt for more details).

The adoption of a standardized and mainlined protocol improves dramatically the portability and the maintainability of the application code.

On Linux kernel, this framework is supported by the kernel 3.10.17_GA – released by Freescale itself along with L3.10.17_1.0.0_IMX6QDLS_BUNDLE BSP – upon which the XELK 2.2.0 is based.

The picture below shows how the system memory is fragmented. The portion of memory used to create a shared area between the two cores – ring buffers – is allocated inside the 256 MByte region used by FreeRTOS.

Memory layout

XELK platform

AXEL Embedded Linux Kit (XELK for short) provides all the necessary components required to set up the developing environment for:

• building the second stage bootloader (U-Boot)
• building and running Linux operating system on AXEL-based systems
• building Linux applications that will run on the target

DAVE Embedded Systems provides all the customization required (in particular at bootloader and Linux kernel levels) to enable customers use the standard i.MX6 development tools for building all the firmware/software components that will run on the target system.

In order to be able to compile the software required in this demo, it is necessary to:

• extract the *.ova inside the SDcard
• install the VM in Virtualbox
• boot the Virtual machine and run the following comand

dvdk@dvdk:~$ sh sdk-mount.sh xelk

Please refer to the XELK Quick Start Guide for further details on XELK.

N.B.: this application note has been tested using XELK 2.0.0.

AMP on AXEL

The following sections describe how to build the software components required to set up asymmetric multi-processing (AMP for short) configuration required to run Linux OS on first Cortex®-A9 core and FreeRTOS on second Cortex®-A9 core.

The latencystat demo is a RPMsg-based application that exploits sophisticated techniques to handle inter-processors communication and synchronization.

Prerequisities

• AXEL Embedded Linux Kit (please refer to XELK Quick Start Guide for further details) version 2.0.0
• access to AXEL git repositories (see below)
• an arm-none-eabi- toolchain for building FreeRTOS

Software components git repositories

The software components for this application note are provided as git repositories, so the user can immediately get access to the source trees and keep these components in sync and up to date with DAVE Embedded Systems repositories.

Assuming that a local repository has not been created, clone the remote AMP git repository:

• cloning the linux git repository:

dvdk@dvdk:~/xelk$ git clone git@git.dave.eu:dave/axel/linux-2.6-imx.git

• cloning the FreeRTOS repository:

dvdk@dvdk:~/xelk$ git clone git@git.dave.eu:dave/axel/freertos.git

• cloning the latencystat repository:

dvdk@dvdk:~/xelk$ git clone git@git.dave.eu:dave/axel/latencystat.git

Building the software components

The following paragraphs describe how to build the software components required for this application. Please note that:

• the standard Linux infrastructure will be used to load the firmware for the second core Linux will start in SMP mode, running on both cores; then CPU1 will be shutdown and freeRTOS firmware will be loaded and run
• the GPIO7_13 pin (JP10.9 of the AXEL EVB-LITE) is toggled during the ISR
• the EIM_D19 signal (available on the R5 resistor on the AXEL EVB-LITE) is the output of the EPIT1 (which is the interrupt source)

The FreeRTOS application programs the EPIT1, which uses 66MHz as time-base (prescaler=1), setting FFFF.FFFF - 0x000F.4240 (1k tick) as the compare value. The timer starts a countdown from 0xFFFF.FFFF and, as it reaches the compare value, it triggers the ISR and toggles the EIM_D19 pin.

The ISR counts the number of ticks after the compare value and saves that information to return it to the Linux application for the histogram calculation.

Build the Linux kernel

Set the correct environment sourcing the iMX6 toolchain:

dvdk@dvdk:~/xelk$ source env.sh 

Enter the linuxdirectory and checkout the amp-feat-2.1.0-amp branch:

dvdk@dvdk:~/xelk/linux-2.6-imx$ git checkout axel-feat-2.1.0-amp

Select the Axel board configuration:

dvdk@dvdk:~/xelk/linux-2.6-imx$ make imx_v7_axel_defconfig

To run this example, Linux kernel must be rebuilt. Thus configure the kernel using imx_v7_axel_defconfig as configuration file and enter the following command line, that changes the default load address of kernel and launches the building of the kernel and device tree images and the kernel modules:

dvdk@dvdk:~/xelk/linux-2.6-imx$ make UIMAGE_LOADADDR=0x18008000 imx6q-xelk-l.dtb uImage modules

The file arch/arm/boot/uImage is the binary image of the kernel that must be used to boot the system, together with the file arch/arm/boot/dts/imx6q-xelk-l.dtb, which is the binary image of the device tree with the XELK hardware configuration.

The kernel modules, for the AMP application, should be installed on root-file system: this can be easily done using the modules_install make parameter

dvdk@dvdk:~/xelk/linux-2.6-imx$ sudo chmod go+rwx /home/dvdk/xelk/rfs/axel-base/lib/modules
dvdk@dvdk:~/xelk/linux-2.6-imx$ make INSTALL_MOD_PATH=/home/dvdk/xelk/rfs/axel-base modules_install

N.B. the previous commands assume that the XELK root file system used is axel-base installed on XELK VM (i.e. /home/dvdk/xelk/rfs/axel-base)

Build FreeRTOS

For building the FreeRTOS, a proper toolchain has to be used, for example, the official arm Linux toolchain ref. Specifically, the example here described was tested with the 7-2017-q4-major, which is available for download here. To install it, just decompress the archive.

Once the toolchain is downloaded and installed, enter the following commands to set up the cross-building environment accordingly:

dvdk@dvdk:~/xelk$ cd freertos
dvdk@dvdk:~/xelk/freertos$ export PATH=<path to arm-eabi toolchain>/bin:$PATH
dvdk@dvdk:~/xelk/freertos$ export ARCH=arm
dvdk@dvdk:~/xelk/freertos$ export CROSS_COMPILE=arm-none-eabi-

Then run make to build the RTOS binary image:

dvdk@dvdk:~/xelk/freertos$ make

This command will build the FreeRTOS kernel and an example application, named Rpmsg-Demo.

When the build process is completede, the example application executable is in the directory output/mx6dq/Rpmsg-Demo/axellite_rev_a. In order to run this application on the target, the Rpmsg-Demo.elf

• has to be copied to the /lib/firmware directory of the target's root file system
• it must be renamed as freertos.

Building latencystat

For building the latercystat example, enter the latecystat directory and then una make:

dvdk@dvdk:~/xelk$ source ~/xelk/env.sh 
dvdk@dvdk:~/xelk$ cd latencystat
dvdk@dvdk:~/xelk/latencystat$ make

The resulting binary must be copied from the building directory to the root file system, e.g. on the /home/root directory of the root-file system.

Running the AMP example application on the target

As stated before, this example shows a sophisticated approach that allows for:

• using a standardized communication channel between the two cores
• exploiting a standardized mechanism to load the firmware of second core

The example performs IRQ latency measurements on FreeRTOS side by using a hardware timer. These measures are collected by the counterpart application running on Linux side and shown on console.

Move the device tree binary load address (the default 0x18000000 conflicts the AMP kernel loadaddr, it must be moved to a higher address) by changing u-boot environment for example as:

U-Boot > setenv fdtaddr 0x1c000000

Moreover, it is possible to force the linux kernel to use only 2 CPUs adding the maxcpus parameter (i.e. on addmisc):

U-Boot > setenv addmisc 'setenv bootargs ${bootargs} vmalloc=400M ${mtdparts}\\;${mtdparts_spi} maxcpus=2'

For a complete uboot enviromente refert to this:

U-Boot > printenv
addcons=setenv bootargs ${bootargs} console=ttymxc2,115200n8
adddisp0=setenv bootargs ${bootargs} video=mxcfb0:dev=lcd,LCD-AM-800480STMQW-TA1,if=RGB666
addhdmi=setenv bootargs ${bootargs} video=mxcfb0:dev=hdmi,1920x1080M@60,bpp=32
addip=setenv bootargs ${bootargs} ip=${ipaddr}:${serverip}:${gatewayip}:${netmask}:${hostname}:${netdev}:off panic=1 fec_mac=${ethaddr}
addlvds0=setenv bootargs ${bootargs}video=mxcfb0:dev=ldb,LDB-AM-800480STMQW-TA1,if=RGB666 ldb=sin0
addlvds1=setenv bootargs ${bootargs}video=mxcfb0:dev=ldb,LDB-AM-800480STMQW-TA1,if=RGB666 ldb=sin1
addmisc=setenv bootargs ${bootargs} vmalloc=400M ${mtdparts}\\;${mtdparts_spi} maxcpus=2
baudrate=115200
bootcmd=mmc dev 0;mmc rescan;if run loadbootscript; then run bootscript; else run mmcboot ; fi; run net_nfs
bootdelay=3
bootfile=xelk/axel-lite/uImage
bootscript=echo Running bootscript from mmc ...; source
disable_giga=1
ethact=FEC
ethaddr=00:50:c2:1e:af:cc
ethprime=FEC0
fdt_high=FFFFFFFF
fdtaddr=0x1c000000
fdtfile=xelk/axel-lite/imx6q-xelk-l.dtb
hostname=xelk
ipaddr=192.168.0.154
load=tftp ${loadaddr} ${uboot}
loadaddr=0x12000000
loadbootscript=fatload mmc 0:1 ${loadaddr} ${script};
loadfdt=tftpboot ${fdtaddr} ${serverip}:${fdtfile}
loadk=tftpboot ${loadaddr} ${serverip}:${bootfile}
mmcargs=setenv bootargs root=${mmcroot}
mmcboot=run mmcargs addcons addmisc adddisp0 mmcloadk; bootm ${loadaddr}
mmcloadk=fatload mmc 0:1 ${loadaddr} uImage
mmcroot=/dev/mmcblk0p2 rootwait rw
module_id#=0x3
mtdids=nand0=gpmi-nand
mtdparts=mtdparts=gpmi-nand:8M(nand-uboot),1M(nand-env1),1M(nand-env2),1M(nand-dtb),1M(nand-spare),8M(nand-kernel),-(nand-ubi)
mtdparts_spi=spi32766.0:1M(spi-uboot),256k(spi-env1),256k(spi-env2),512k(spi-dtb),6M(spi-kernel),-(spi-free)
net_nfs=run loadk loadfdt nfsargs addip addcons addmisc adddisp0; bootm ${loadaddr} - ${fdtaddr}
netdev=eth0
netmask=255.255.255.0
nfsargs=setenv bootargs root=/dev/nfs rw nfsroot=${serverip}:${rootpath},v3,tcp
rootpath=/home/dvdk/xelk/rfs/axel-base
script=boot.scr
serverip=192.168.0.153
spi_update=sf probe; sf erase 0 80000;sf write ${loadaddr} 400 80000
stderr=serial
stdin=serial
stdout=serial
uboot=axel/u-boot.imx
ver=U-Boot 2013.04 (Nov 21 2014 - 16:21:37)-xelk-2.0.0

Environment size: 2303/262139 bytes
U-Boot >

Once all the components are built, please boot the system and launch the following commands:./

echo "now running with $(cat /proc/cpuinfo | grep processor | wc -l) processors"

echo "start remote proc AMP"

modprobe virtio
modprobe virtio_ring
modprobe virtio_rpmsg_bus
modprobe remoteproc
modprobe mx6_remoteproc
modprobe rpmsg_freertos_statistic

echo "everything done"
echo "now Linux is running with $(cat /proc/cpuinfo | grep processor | wc -l) processors"

It is worth remembering that the previous commands load the required drivers to run the AMP configuration. Specifically, they:

• Release one of the Cortex A9 cores; the released core will be used to run the FreeRTOS-based example application
• Set up the RPmsg communication channel between the Linux kernel and the FreeRTOS kernel
• Load the code of the FreeRTOS-based example application into SDRAM memory (the code is retrieved from the /lib/freertos ELF file)
• Set up the devoted core to run the FreeRTOS-based example application.

On Linux side, run the latencystat application as shown below. latencystat application will communicate with the firmware running on the FreeRTOS-dedicated core through RPMsg. The typical output will look like this:

root@axel-lite:~# ./latencystat -b
Linux FreeRTOS AMP Demo.
   0: Command 0 ACKed
   1: Command 1 ACKed
Waiting for samples...
   2: Command 2 ACKed
   3: Command 3 ACKed
   4: Command 4 ACKed
-----------------------------------------------------------
Histogram Bucket Values:
        Bucket 393 ns (26 ticks) had 1 frequency
        Bucket 409 ns (27 ticks) had 475 frequency
        Bucket 424 ns (28 ticks) had 1058 frequency
        Bucket 439 ns (29 ticks) had 57 frequency
        Bucket 681 ns (45 ticks) had 1 frequency
-----------------------------------------------------------
Histogram Data:
        min: 393 ns (26 ticks)
        avg: 409 ns (27 ticks)
        max: 681 ns (45 ticks)
        out of range: 0
        total samples: 1592
-----------------------------------------------------------

This application is extremely useful for measuring interrupt latency and for verifying how it is affected by relevant parameters such as the CPU load on the first core, memory bandwidth utilization, the amount/rate of interrupts to be processed by the first core, operating frequency, etc.

For instance, the previous data refer to the following configuration:

• governor: userspace
• nominal operating frequency: 800 MHz (actual: 792 MHz)
• core #0 (running Linux) lightly loaded with minimal I/O activity.

In general, it is strongly recommended to characterize the latency on the basis of a realistic testbed that is as close as possible to the actual use case. It is also worth remembering that, in case of latency does not satisfy the application's real-time requirements, it is possible to adjust the arbitration priorities of processor's interconnect subsystem to reduce it.