BELK-AN-007: Asymmetric Multiprocessing (AMP) on Bora/BoraX with OpenAMP

Info Box Applies to Bora Applies to BORA Xpress Applies to BORA Application Notes Applies to BORA Xpress Application Notes

This application note was validated against specific versions of the kit only. It may not work with other versions. Supported versions are listed in the History section.

The implementation described in this document applies do the BELK/BXELK versions indicated in the History section. For previous releases, please refer to this page.

History

Version	Date	BELK/BXELK version	Notes
0.9.0	September 2017	4.0.0 / 2.0.0	First draft

Introduction

Asymmetric Multi Processing (AMP) allows a multiprocessor system to run multiple Operating Systems (OS) that are independent of each other. In other words, each CPU has its own private memory space, which contains the OS and the applications that are to run on that CPU. In addition, there can be some shared memory space that is used for multiprocessor communication. This is contrasted with Symmetric Multiprocessing (SMP), in which one OS runs on multiple CPUs using a public shared memory space. Thanks to AMP, developers can use open-source Linux and FreeRTOS operating systems and the RPMsg Inter Processor Communication (IPC) framework between the Zynq's two high-performance ARM® Cortex™-A9 processors to quickly implement applications that need to deliver deterministic, real-time responsiveness for markets such as automotive, industrial and others with similar requirements. For further information, please refer to http://www.wiki.xilinx.com/Multi-OS+Support+%28AMP+%26+Hypervisor%29

BELK/BXELK AMP demos are based on OpenAMP Framework for Zynq Devices (see UG1186). Open Asymmetric Multi-processing (OpenAMP) is a framework providing the software components needed to enable the development of software applications for asymmetric multi-processing (AMP) systems. The Xilinx implementation of OpenAMP framework provides the following for Zynq-7000 All Programmable (AP) SoC devices:

• The remoteproc , RPMsg , and virtIO components that are used for a Linux master or a bare-metal remote configuration.
• Proxy infrastructure and demos that showcase the ability of a proxy on a master processor running Linux on the ARM processor unit (APU) to handle printf , scanf , open , close , read , and write calls from a bare-metal OS-based remote contexts running on the remote processor unit (RPU).

Build the AN-BELK-007

Yocto build

To build the components for the AN-BELK-007, the users can build the Yocto BSP as described here. Some changes must be made to build Yocto for AN-BELK-007. After initializing the build environment as described here, user must modify the conf/local.conf build configuration to match the bora-amp machine:

MACHINE = "bora-amp"

Then build the bora-image-devel image as described here.

Separated components build

Linux Kernel

To build Linux Kernel for AN-BELK-007 follow the instructions described here to setup the build environment and run the following commands in the source tree directory:

make bora_defconfig
make UIMAGE_LOADADDR=0x10008000 uImage bora-amp.dtb

Demo Applications

To build the AN-BELK-007 linux userspace demo applications run the following commands (the following is for the rpmsg-echo-test):

dvdk@dvdk-vm:~$ . ~/bora/sdk/latest/environment-setup-cortexa9hf-neon-xilinx-linux-gnueabi
dvdk@dvdk-vm:~$ git clone https://github.com/Xilinx/meta-openamp -b rel-v2017.2
Cloning into 'meta-openamp'...
remote: Counting objects: 378, done.
remote: Compressing objects: 100% (163/163), done.
remote: Total 378 (delta 189), reused 378 (delta 189), pack-reused 0
Receiving objects: 100% (378/378), 58.85 KiB | 0 bytes/s, done.
Resolving deltas: 100% (189/189), done.
Checking connectivity... done.
dvdk@dvdk-vm:~$ cd meta-openamp/recipes-openamp/rpmsg-examples/rpmsg-echo-test/
dvdk@dvdk-vm:~/meta-openamp/recipes-openamp/rpmsg-examples/rpmsg-echo-test$ make
arm-xilinx-linux-gnueabi-gcc  -march=armv7-a -marm -mfpu=neon  -mfloat-abi=hard -mcpu=cortex-a9 --sysroot=/home/dvdk/bora/sdk/belk-4.0.0/sysroots/cortexa9hf-neon-xilinx-linux-gnueabi -c -O2 -pipe -g -feliminate-unused-debug-types -o echo_test.o echo_test.c
arm-xilinx-linux-gnueabi-gcc  -march=armv7-a -marm -mfpu=neon  -mfloat-abi=hard -mcpu=cortex-a9 --sysroot=/home/dvdk/bora/sdk/belk-4.0.0/sysroots/cortexa9hf-neon-xilinx-linux-gnueabi -Wl,-O1 -Wl,--hash-style=gnu -Wl,--as-needed -o echo_test echo_test.o 
dvdk@dvdk-vm:~/meta-openamp/recipes-openamp/rpmsg-examples/rpmsg-echo-test$ ll
total 60
drwxrwxr-x 2 dvdk dvdk  4096 Sep 21 09:13 ./
drwxrwxr-x 5 dvdk dvdk  4096 Sep 21 09:00 ../
-rwxrwxr-x 1 dvdk dvdk 18512 Sep 21 09:13 echo_test*
-rw-rw-r-- 1 dvdk dvdk  4241 Sep 21 09:00 echo_test.c
-rw-rw-r-- 1 dvdk dvdk 13932 Sep 21 09:13 echo_test.o
-rw-rw-r-- 1 dvdk dvdk  1686 Sep 21 09:00 LICENSE
-rw-rw-r-- 1 dvdk dvdk   237 Sep 21 09:00 Makefile

Copy the echo_test executable binary into target rootfilesystem.

FreeRTOS Firmware

FreeRTOS Firmware for the AN-BELK-007 demos can be buildt starting from the BELK/BXELK Vivado project described here, or from a custom BORA/BORAX Vivado project.

Follow these steps with the Vivado project already opened:

• launch the SDK session, clicking on File -> Launch SDK
• once the Xilinx SDK is ready, perform the following operations from the GUI:
  • Click on File -> New -> Application Project
  • select the Project Name: openamp_echo (or openamp_matrix_mult or openamp_rpc for the other demos)
  • select OS Platform: freertos901_xilinx
  • select Processor: ps7_cortexa9_1
  • Click Next
  • Select Template: OpenAMP echo-test (or OpenAMP matrix multiplication Demo or OpenAMP RPC Demo for the other demos)
  • Click on Finish
• On Project Explorer tab right click on the openamp_echo_bsp project and select Board Support Package Settings
  • under Overview -> freertos901_xilinx -> openamp set WITH_PROXY parameter to FALSE for echo and matrix demos, set to TRUE for proxy demo
  • under drivers -> ps7_cortexa9_1 add -DUSE_AMP=1 to extra_compiler_flags parameter
  • close settings by clicking OK button
• rebuild project by right clicking on the project name in Project Explorer and selecting first Clean Project and then Build Project
• the FreeRTOS Firmware demo project (ELF file) is then built and available under the Debug directory of the project
• copy the openamp_echo.elf (or openamp_matrix_mult.elf or openamp_rpc.elf for the other demos) into the target root filesystem under the /lib/firmware/ directory.

Run AN-BELK-007 demos

The AN-BELK-007 demos can be executed on BELK/BXELK kit using the pre-built images release along with the kit, or using Yocto build system as described here.

Create a bootable microSD card as described here.

Boot the system via NFS as described here, loading the AN-BELK-007 kernel, devicetree and RFS.

Once the system is booted into linux prompt users are able to run one of the tree examples provided with the AN-BELK-007:

• Echo Test in Linux Master and FreeRTOS Remote
• Matrix Multiplication for Linux Master and FreeRTOS Remote
• Proxy Application for Linux Masters and FreeRTOS Remote

Echo demo

Here are instructions to run echo test demo :

root@bora-amp:~# echo openamp_echo.elf > /sys/class/remoteproc/remoteproc0/firmware
root@bora-amp:~# echo start > /sys/class/remoteproc/remoteproc0/state
[  246.809132] remoteproc remoteproc0: powering up remoteproc@0
[  246.832873] remoteproc remoteproc0: Booting fw image openamp_echo.elf, size 583420
[  246.840650] remoteproc remoteproc0: registered virtio0 (type 7)
[  246.860423] virtio_rpmsg_bus virtio0: rpmsg host is online
[  246.879393] CPU1: shutdown
[  246.930106] remoteproc remoteproc0: remote processor remoteproc@0 is now up
[  246.937047] virtio_rpmsg_bus virtio0: creating channel rpmsg-openamp-demo-channel addr 0x1
root@bora-amp:~# modprobe rpmsg_user_dev_driver
[  251.607015] rpmsg_user_dev_driver: loading out-of-tree module taints kernel.
[  251.616325] rpmsg_user_dev_driver virtio0:rpmsg-openamp-demo-channel: rpmsg_user_dev_rpmsg_drv_probe
[  251.631722] rpmsg_user_dev_driver virtio0:rpmsg-openamp-demo-channel: new channel: 0x400 -> 0x1!
root@bora-amp:~# echo_test

 Echo test start

 Open rpmsg dev!
[  256.285856] rpmsg_user_dev_driver virtio0:rpmsg-openamp-demo-channel: Sent init_msg to target 0x1.

 ****************************************
 Please enter command and press enter key
 ****************************************
 1 - Send data to remote core, retrieve the echo and validate its integrity ..
 2 - Quit this application ..
 CMD>1

 sending payload number 0 of size 9
echo test: sent : 9
 received payload number 0 of size 9

 sending payload number 2 of size 10
echo test: sent : 10
 received payload number 2 of size 10

......

 sending payload number 471 of size 479
echo test: sent : 479
 received payload number 471 of size 479

 **************************************

 Test Results: Error count = 0

 **************************************

 ****************************************
 Please enter command and press enter key
 ****************************************
 1 - Send data to remote core, retrieve the echo and validate its integrity ..
 2 - Quit this application ..
 CMD>2
[  264.125688] rpmsg_user_dev_driver virtio0:rpmsg-openamp-demo-channel: Sending shutdown message.

[  264.134523] virtio_rpmsg_bus virtio0: destroying channel rpmsg-openamp-demo-channel addr 0x1
 Quitting application ..
 Echo test end
[  264.144173] rpmsg_user_dev_driver virtio0:rpmsg-openamp-demo-channel: Removing rpmsg user dev.
[  264.156716] rpmsg_user_dev_driver rpmsg0: Releasing rpmsg user dev device.
root@bora-amp:~# modprobe -r rpmsg_user_dev_driver
root@bora-amp:~# echo stop > /sys/class/remoteproc/remoteproc0/state
[  275.510501] remoteproc remoteproc0: stopped remote processor remoteproc@0

Matrix multiply demo

Here are instructions to run matrix multiply demo :

root@bora-amp:~# echo openamp_matrix_mult.elf > /sys/class/remoteproc/remoteproc0/firmware
root@bora-amp:~# echo start > /sys/class/remoteproc/remoteproc0/state
[  293.159117] remoteproc remoteproc0: powering up remoteproc@0
[  293.183492] remoteproc remoteproc0: Booting fw image openamp_matrix_mult.elf, size 584436
[  293.192369] virtio_rpmsg_bus virtio0: rpmsg host is online
[  293.197834] remoteproc remoteproc0: registered virtio0 (type 7)
[  293.239339] CPU1: shutdown
[  293.270113] remoteproc remoteproc0: remote processor remoteproc@0 is now up
[  293.277042] virtio_rpmsg_bus virtio0: creating channel rpmsg-openamp-demo-channel addr 0x1
root@bora-amp:~# modprobe rpmsg_user_dev_driver
[  296.803836] rpmsg_user_dev_driver virtio0:rpmsg-openamp-demo-channel: rpmsg_user_dev_rpmsg_drv_probe
[  296.819696] rpmsg_user_dev_driver virtio0:rpmsg-openamp-demo-channel: new channel: 0x400 -> 0x1!
root@bora-amp:~# mat_mul_demo

 Matrix multiplication demo start

 Open rpmsg dev!
[  301.987135] rpmsg_user_dev_driver virtio0:rpmsg-openamp-demo-channel: Sent init_msg to target 0x1.

 Creating ui_thread and compute_thread ...

 ****************************************
 Please enter command and press enter key
 ****************************************
 1 - Generates random 6x6 matrices and transmits them to remote core over rpmsg ..
 2 - Quit this application ..
 CMD>1

 Compute thread unblocked ..
 The compute thread is now blocking ona read() from rpmsg device

 Generating random matrices now ...

 Master : Linux : Input matrix 0

 2  4  9  8  3  1
 4  7  1  2  1  2
 5  1  6  0  3  8
 7  7  2  8  6  2
 3  3  2  9  6  7
 0  8  2  1  6  7

 Master : Linux : Input matrix 1

 4  3  4  7  5  7
 9  0  1  7  3  4
 5  2  4  9  1  2
 3  6  5  5  5  1
 5  7  0  7  0  6
 4  4  1  0  1  9

 Writing generated matrices to rpmsg rpmsg device, 296 bytes written ..

 Received results! - 148 bytes from rpmsg device (transmitted from remote context)

 Master : Linux : Printing results
 132  97  89  184  72  83
 103  41  39  103  54  84
 106  80  53  117  42  141
 163  123  85  198  100  143
 134  137  75  147  78  145
 143  80  28  121  38  136

 ****************************************
 Please enter command and press enter key
 ****************************************
 1 - Generates random 6x6 matrices and transmits them to remote core over rpmsg ..
 2 - Quit this application ..
 CMD>2

 Quitting application ..
[  305.822764] rpmsg_user_dev_driver virtio0:rpmsg-openamp-demo-channel: Sending shutdown message.


[  305.837041] virtio_rpmsg_bus virtio0: destroying channel rpmsg-openamp-demo-channel addr 0x1
 Quitting application ..
 Matrix multiply application end
[  305.846823] rpmsg_user_dev_driver virtio0:rpmsg-openamp-demo-channel: Removing rpmsg user dev.
[  305.861495] rpmsg_user_dev_driver rpmsg0: Releasing rpmsg user dev device.
root@bora-amp:~#
root@bora-amp:~# modprobe -r rpmsg_user_dev_driver
root@bora-amp:~# echo stop > /sys/class/remoteproc/remoteproc0/state
[  315.160499] remoteproc remoteproc0: stopped remote processor remoteproc@0

Proxy demo

Here are instructions to run proxy demo :

root@bora-amp:~# proxy_app -f /lib/firmware/openamp_rpc.elf

Master>Loading remote firmware
[  476.526797] remoteproc remoteproc0: powering up remoteproc@0
[  476.536009] remoteproc remoteproc0: Booting fw image image_rpc_demo, size 656576
[  476.544880] virtio_rpmsg_bus virtio0: rpmsg host is online
[  476.550441] remoteproc remoteproc0: registered virtio0 (type 7)
[  476.589353] CPU1: shutdown
[  476.620128] remoteproc remoteproc0: remote processor remoteproc@0 is now up
[  476.627103] virtio_rpmsg_bus virtio0: creating channel rpmsg-openamp-demo-channel addr 0x1

Master>Create rpmsg proxy device
[  476.659720] rpmsg_proxy_dev_rpmsg virtio0:rpmsg-openamp-demo-channel: rpmsg_proxy_dev_rpmsg_drv_probe
[  476.670411] rpmsg_proxy_dev_rpmsg virtio0:rpmsg-openamp-demo-channel: new channel: 0x400 -> 0x1!

Master>Opening rpmsg proxy device
[  476.681017] rpmsg_proxy_dev_rpmsg virtio0:rpmsg-openamp-demo-channel: Sent init_msg to target 0x1.

Master>RPC service started !!

Remote>FreeRTOS Remote Procedure Call (RPC) Demonstration

Remote>***************************************************

Remote>Rpmsg based retargetting to proxy initialized..

Remote>FileIO demo ..

Remote>Creating a file on master and writing to it..

Remote>Opened file 'remote.file' with fd = -1

Remote>Wrote to fd = -1, size = -1, content = This is a test string being written to file..

Remote>Closed fd = -1

Remote>Reading a file on master and displaying its contents..

Remote>Opened file 'remote.file' with fd = 4

Remote>Read from fd = 4, size = 45, printing contents below .. This is a test string being written to file..¥

Remote>Closed fd = 4

Remote>Remote firmware using scanf and printf ..

Remote>Scanning user input from master..

Remote>Enter name
Dave

Remote>Enter age
19

Remote>Enter value for pi
3.1415926535897932384626433832795

Remote>User name = 'Dave'

Remote>User age = '19'

Remote>User entered value of pi = '3.141593'

Remote>Repeat demo ? (enter yes or no)
no

Remote>RPC retargetting quitting ...

Remote> Firmware's rpm[  530.949410] rpmsg_proxy_dev_rpmsg virtio0:rpmsg-openamp-demo-channel: Sending terminate message.
sg-openamp-demo-channel going down!

Master>RPC service exiting !!
Master> sending shutdown signal.
[  530.980772] rpmsg_proxy_dev_rpmsg virtio0:rpmsg-openamp-demo-channel: Removing rpmsg proxy dev.
[  530.990661] rpmsg_proxy_dev_driver rpmsg_proxy0: Releasing rpmsg proxy dev device.
[  531.030720] remoteproc remoteproc0: stopped remote processor remoteproc@0
root@bora-amp:~#