XELK-AN-001: Asymmetric Multiprocessing (AMP) on Axel – Linux + FreeRTOS
Info Box
|
![]() |
This application note was validated against specific versions of the kit only. It may not work with other versions. Supported versions are listed in the History section. | ![]() |
Contents
History[edit | edit source]
Version | Date | XELK version | Notes |
---|---|---|---|
0.9.2 | Feb 2015 | 2.0.0 | Update for XELK 2.0.0 release |
0.9.3 | April 2015 | 2.0.0 | Minor fixes |
1.0.0 | April 2018 | 2.1.0 | Update for amp-feat-2.1.0-amp branch |
1.0.1 | October 2018 | 2.1.0 | Minor improvements |
1.0.2 | October 2018 | 2.1.0 | Added some details about latencystat test application
|
Introduction[edit | edit source]
This application note describe how to build the software components required to set up asymmetric multi-processing (AMP for short).
The following configuration required to run Linux OS on first Cortex®-A9 core and FreeRTOS on second Cortex®-A9 core of the Freescale i.MX6 SOC.
- N.B. For the sake of simplicity, in this application note, we are going to use two of the four core inside the iMX6Q cpu. Other configuration are possible but are not relevant for this demo.
The latencystat demo is a RPMsg-based application that exploits sophisticated techniques to handle inter-processors communication and synchronization.
Asymmetric Multiprocessing[edit | edit source]
Thanks to latest technological improvements, multicore processors are becoming popular in embedded world too. These architectures allows the implementation of processing schemes that were not feasible with traditional single-core CPUs. Among these, one of the most interesting is asymmetric multiprocessing (AMP for short). This configuration permits to address several requirements that the embedded system developers struggle to handle in case of single-core systems.
This application note describes in detail the implementation of Linux/FreeRTOS asymmetric multiprocessing configuration on DAVE Embedded Systems AXEL LITE SoM. This configuration is a typical example about how to leverage AMP flexibility to combine, on one single piece of silicon, the versatility of Linux o.s. for general purpose computation, connectivity and HMI and the determinism of an RTOS to satisfy real-time constraints. Since AXEL family products are all based on Freescale i.MX6 processors, what is described here applies to AXEL ULTRA too.
As known, AMP allows a multicore system to run simultaneously [1] multiple Operating Systems (OS) that are independent of each other. In other words, each core has its own private memory space, which contains the OS and the applications that are to run on that core. In addition, there can be some shared memory space that is used for inter-core communication. This is in contrast to Symmetric Multiprocessing (SMP), in which one OS runs on multiple cores using a public shared memory space.
Thanks to AMP, developers can use open-source Linux and FreeRTOS operating systems and the RPMsg Inter Processor Communication (IPC) framework to quickly implement applications that need to deliver deterministic, real-time responsiveness for markets such as automotive, industrial and others with similar requirements, while preserving the openness of Linux.
The following picture depicts the structure of the system.
Core #0:
- takes care of boot process
- once Linux gets control of the processor, initializes core #1 in SMP mode
- stops core #1 and switches it to AMP mode
- loads binary image of FreeRTOS that is then executed by core #1
[1] This is true simultaneity because multiple instructions are executed at the same time.
Inter-core communication[edit | edit source]
Inter-core communication is based on RPMsg framework (please refer to http://omappedia.org/wiki/Category:RPMsg and https://www.kernel.org/doc/Documentation/rpmsg.txt for more details).
The adoption of a standardized and mainlined protocol improves dramatically the portability and the maintainability of the application code.
On Linux kernel, this framework is supported by the kernel 3.10.17_GA – released by Freescale itself along with L3.10.17_1.0.0_IMX6QDLS_BUNDLE BSP – upon which the XELK 2.2.0 is based.
The picture below shows how the system memory is fragmented. The portion of memory used to create a shared area between the two cores – ring buffers – is allocated inside the 256 MByte region used by FreeRTOS.
XELK platform[edit | edit source]
AXEL Embedded Linux Kit (XELK for short) provides all the necessary components required to set up the developing environment for:
- building the second stage bootloader (U-Boot)
- building and running Linux operating system on AXEL-based systems
- building Linux applications that will run on the target
DAVE Embedded Systems provides all the customization required (in particular at bootloader and Linux kernel levels) to enable customers use the standard i.MX6 development tools for building all the firmware/software components that will run on the target system.
In order to be able to compile the software required in this demo, it is necessary to:
- extract the *.ova inside the SDcard
- install the VM in Virtualbox
- boot the Virtual machine and run the following comand
dvdk@dvdk:~$ sh sdk-mount.sh xelk
Please refer to the XELK Quick Start Guide for further details on XELK.
N.B.: this application note has been tested using XELK 2.0.0.
AMP on AXEL[edit | edit source]
The following sections describe how to build the software components required to set up asymmetric multi-processing (AMP for short) configuration required to run Linux OS on first Cortex®-A9 core and FreeRTOS on second Cortex®-A9 core.
The latencystat
demo is a RPMsg-based application that exploits sophisticated techniques to handle inter-processors communication and synchronization.
Prerequisities[edit | edit source]
- AXEL Embedded Linux Kit (please refer to XELK Quick Start Guide for further details) version 2.0.0
- access to AXEL git repositories (see below)
- an
arm-none-eabi-
toolchain for building FreeRTOS
Software components git repositories[edit | edit source]
The software components for this application note are provided as git repositories, so the user can immediately get access to the source trees and keep these components in sync and up to date with DAVE Embedded Systems repositories.
Assuming that a local repository has not been created, clone the remote AMP git repository:
- cloning the
linux
git repository:
dvdk@dvdk:~/xelk$ git clone git@git.dave.eu:dave/axel/linux-2.6-imx.git
- cloning the
FreeRTOS
repository:
dvdk@dvdk:~/xelk$ git clone git@git.dave.eu:dave/axel/freertos.git
- cloning the
latencystat
repository:
dvdk@dvdk:~/xelk$ git clone git@git.dave.eu:dave/axel/latencystat.git
Building the software components[edit | edit source]
The following paragraphs describe how to build the software components required for this application. Please note that:
- the standard Linux infrastructure will be used to load the firmware for the second core Linux will start in SMP mode, running on both cores; then CPU1 will be shutdown and freeRTOS firmware will be loaded and run
- the GPIO7_13 pin (JP10.9 of the AXEL EVB-LITE) is toggled during the ISR
- the EIM_D19 signal (available on the R5 resistor on the AXEL EVB-LITE) is the output of the EPIT1 (which is the interrupt source)
The FreeRTOS application programs the EPIT1, which uses 66MHz as time-base (prescaler=1), setting FFFF.FFFF - 0x000F.4240 (1k tick) as the compare value. The timer starts a countdown from 0xFFFF.FFFF and, as it reaches the compare value, it triggers the ISR and toggles the EIM_D19 pin.
The ISR counts the number of ticks after the compare value and saves that information to return it to the Linux application for the histogram calculation.
Build the Linux kernel[edit | edit source]
Set the correct environment sourcing the iMX6 toolchain:
dvdk@dvdk:~/xelk$ source env.sh
Enter the linux
directory and checkout the amp-feat-2.1.0-amp
branch:
dvdk@dvdk:~/xelk/linux-2.6-imx$ git checkout axel-feat-2.1.0-amp
Select the Axel board configuration:
dvdk@dvdk:~/xelk/linux-2.6-imx$ make imx_v7_axel_defconfig
To run this example, Linux kernel must be rebuilt. Thus configure the kernel using imx_v7_axel_defconfig as configuration file and enter the following command line, that changes the default load address of kernel and launches the building of the kernel and device tree images and the kernel modules:
dvdk@dvdk:~/xelk/linux-2.6-imx$ make UIMAGE_LOADADDR=0x18008000 imx6q-xelk-l.dtb uImage modules
The file arch/arm/boot/uImage
is the binary image of the kernel that must be used to boot the system, together with the file arch/arm/boot/dts/imx6q-xelk-l.dtb
, which is the binary image of the device tree with the XELK hardware configuration.
The kernel modules, for the AMP application, should be installed on root-file system: this can be easily done using the modules_install
make parameter
dvdk@dvdk:~/xelk/linux-2.6-imx$ sudo chmod go+rwx /home/dvdk/xelk/rfs/axel-base/lib/modules dvdk@dvdk:~/xelk/linux-2.6-imx$ make INSTALL_MOD_PATH=/home/dvdk/xelk/rfs/axel-base modules_install
N.B. the previous commands assume that the XELK root file system used is axel-base
installed on XELK VM (i.e. /home/dvdk/xelk/rfs/axel-base
)
Build FreeRTOS[edit | edit source]
For building the FreeRTOS, a proper toolchain has to be used, for example, the official arm Linux toolchain ref. Specifically, the example here described was tested with the 7-2017-q4-major
, which is available for download here. To install it, just decompress the archive.
Once the toolchain is downloaded and installed, enter the following commands to set up the cross-building environment accordingly:
dvdk@dvdk:~/xelk$ cd freertos dvdk@dvdk:~/xelk/freertos$ export PATH=<path to arm-eabi toolchain>/bin:$PATH dvdk@dvdk:~/xelk/freertos$ export ARCH=arm dvdk@dvdk:~/xelk/freertos$ export CROSS_COMPILE=arm-none-eabi-
Then run make
to build the RTOS binary image:
dvdk@dvdk:~/xelk/freertos$ make
This command will build the FreeRTOS kernel and an example application, named Rpmsg-Demo
.
When the build process is completede, the example application executable is in the directory output/mx6dq/Rpmsg-Demo/axellite_rev_a
. In order to run this application on the target, the Rpmsg-Demo.elf
- has to be copied to the
/lib/firmware
directory of the target's root file system - it must be renamed as
freertos
.
Building latencystat[edit | edit source]
For building the latercystat
example, enter the latecystat
directory and then una make
:
dvdk@dvdk:~/xelk$ source ~/xelk/env.sh dvdk@dvdk:~/xelk$ cd latencystat dvdk@dvdk:~/xelk/latencystat$ make
The resulting binary must be copied from the building directory to the root file system, e.g. on the /home/root
directory of the root-file system.
Running the AMP example application on the target[edit | edit source]
As stated before, this example shows a sophisticated approach that allows for:
- using a standardized communication channel between the two cores
- exploiting a standardized mechanism to load the firmware of second core
The example performs IRQ latency measurements on FreeRTOS side by using a hardware timer. These measures are collected by the counterpart application running on Linux side and shown on console.
Move the device tree binary load address (the default 0x18000000 conflicts the AMP kernel loadaddr, it must be moved to a higher address) by changing u-boot environment for example as:
U-Boot > setenv fdtaddr 0x1c000000
Moreover, it is possible to force the linux kernel to use only 2 CPUs adding the maxcpus
parameter (i.e. on addmisc):
U-Boot > setenv addmisc 'setenv bootargs ${bootargs} vmalloc=400M ${mtdparts}\\;${mtdparts_spi} maxcpus=2'
For a complete uboot enviromente refert to this:
U-Boot > printenv addcons=setenv bootargs ${bootargs} console=ttymxc2,115200n8 adddisp0=setenv bootargs ${bootargs} video=mxcfb0:dev=lcd,LCD-AM-800480STMQW-TA1,if=RGB666 addhdmi=setenv bootargs ${bootargs} video=mxcfb0:dev=hdmi,1920x1080M@60,bpp=32 addip=setenv bootargs ${bootargs} ip=${ipaddr}:${serverip}:${gatewayip}:${netmask}:${hostname}:${netdev}:off panic=1 fec_mac=${ethaddr} addlvds0=setenv bootargs ${bootargs}video=mxcfb0:dev=ldb,LDB-AM-800480STMQW-TA1,if=RGB666 ldb=sin0 addlvds1=setenv bootargs ${bootargs}video=mxcfb0:dev=ldb,LDB-AM-800480STMQW-TA1,if=RGB666 ldb=sin1 addmisc=setenv bootargs ${bootargs} vmalloc=400M ${mtdparts}\\;${mtdparts_spi} maxcpus=2 baudrate=115200 bootcmd=mmc dev 0;mmc rescan;if run loadbootscript; then run bootscript; else run mmcboot ; fi; run net_nfs bootdelay=3 bootfile=xelk/axel-lite/uImage bootscript=echo Running bootscript from mmc ...; source disable_giga=1 ethact=FEC ethaddr=00:50:c2:1e:af:cc ethprime=FEC0 fdt_high=FFFFFFFF fdtaddr=0x1c000000 fdtfile=xelk/axel-lite/imx6q-xelk-l.dtb hostname=xelk ipaddr=192.168.0.154 load=tftp ${loadaddr} ${uboot} loadaddr=0x12000000 loadbootscript=fatload mmc 0:1 ${loadaddr} ${script}; loadfdt=tftpboot ${fdtaddr} ${serverip}:${fdtfile} loadk=tftpboot ${loadaddr} ${serverip}:${bootfile} mmcargs=setenv bootargs root=${mmcroot} mmcboot=run mmcargs addcons addmisc adddisp0 mmcloadk; bootm ${loadaddr} mmcloadk=fatload mmc 0:1 ${loadaddr} uImage mmcroot=/dev/mmcblk0p2 rootwait rw module_id#=0x3 mtdids=nand0=gpmi-nand mtdparts=mtdparts=gpmi-nand:8M(nand-uboot),1M(nand-env1),1M(nand-env2),1M(nand-dtb),1M(nand-spare),8M(nand-kernel),-(nand-ubi) mtdparts_spi=spi32766.0:1M(spi-uboot),256k(spi-env1),256k(spi-env2),512k(spi-dtb),6M(spi-kernel),-(spi-free) net_nfs=run loadk loadfdt nfsargs addip addcons addmisc adddisp0; bootm ${loadaddr} - ${fdtaddr} netdev=eth0 netmask=255.255.255.0 nfsargs=setenv bootargs root=/dev/nfs rw nfsroot=${serverip}:${rootpath},v3,tcp rootpath=/home/dvdk/xelk/rfs/axel-base script=boot.scr serverip=192.168.0.153 spi_update=sf probe; sf erase 0 80000;sf write ${loadaddr} 400 80000 stderr=serial stdin=serial stdout=serial uboot=axel/u-boot.imx ver=U-Boot 2013.04 (Nov 21 2014 - 16:21:37)-xelk-2.0.0 Environment size: 2303/262139 bytes U-Boot >
Once all the components are built, please boot the system and launch the following commands:./
echo "now running with $(cat /proc/cpuinfo | grep processor | wc -l) processors" echo "start remote proc AMP" modprobe virtio modprobe virtio_ring modprobe virtio_rpmsg_bus modprobe remoteproc modprobe mx6_remoteproc modprobe rpmsg_freertos_statistic echo "everything done" echo "now Linux is running with $(cat /proc/cpuinfo | grep processor | wc -l) processors"
It is worth remembering that the previous commands load the required drivers to run the AMP configuration. Specifically, they:
- Release one of the Cortex A9 cores; the released core will be used to run the FreeRTOS-based example application
- Set up the RPmsg communication channel between the Linux kernel and the FreeRTOS kernel
- Load the code of the FreeRTOS-based example application into SDRAM memory (the code is retrieved from the
/lib/freertos
ELF file) - Set up the devoted core to run the FreeRTOS-based example application.
On Linux side, run the latencystat
application as shown below. latencystat
application will communicate with the firmware running on the FreeRTOS-dedicated core through RPMsg. The typical output will look like this:
root@axel-lite:~# ./latencystat -b Linux FreeRTOS AMP Demo. 0: Command 0 ACKed 1: Command 1 ACKed Waiting for samples... 2: Command 2 ACKed 3: Command 3 ACKed 4: Command 4 ACKed ----------------------------------------------------------- Histogram Bucket Values: Bucket 393 ns (26 ticks) had 1 frequency Bucket 409 ns (27 ticks) had 475 frequency Bucket 424 ns (28 ticks) had 1058 frequency Bucket 439 ns (29 ticks) had 57 frequency Bucket 681 ns (45 ticks) had 1 frequency ----------------------------------------------------------- Histogram Data: min: 393 ns (26 ticks) avg: 409 ns (27 ticks) max: 681 ns (45 ticks) out of range: 0 total samples: 1592 -----------------------------------------------------------
This application is extremely useful for measuring interrupt latency and for verifying how it is affected by relevant parameters such as the CPU load on the first core, memory bandwidth utilization, the amount/rate of interrupts to be processed by the first core, operating frequency, etc.
For instance, the previous data refer to the following configuration:
- governor:
userspace
- nominal operating frequency: 800 MHz (actual: 792 MHz)
- core #0 (running Linux) lightly loaded with minimal I/O activity.
In general, it is strongly recommended to characterize the latency on the basis of a realistic testbed that is as close as possible to the actual use case. It is also worth remembering that, in case of latency does not satisfy the application's real-time requirements, it is possible to adjust the arbitration priorities of processor's interconnect subsystem to reduce it.