DESK-XZ7-L-AN-0002: Asymmetric Multiprocessing (AMP) on Bora with OpenAMP

Info Box Applies to BORA Application Notes

This application note has been validated using the kit version in the History table.

History
Issue Date	Notes
2024/01/30	DESK-XZ7-L-1.x.x release

Introduction

Asymmetric Multi Processing (AMP) allows a multiprocessor system to run multiple Operating Systems (OS) that are independent of each other. In other words, each CPU has its own private memory space, which contains the OS and the applications that are to run on that CPU. In addition, there can be some shared memory space that is used for multiprocessor communication. This is contrasted with Symmetric Multiprocessing (SMP), in which one OS runs on multiple CPUs using a public shared memory space. Thanks to AMP, developers can use open-source Linux and FreeRTOS operating systems and the RPMsg Inter Processor Communication (IPC) framework between the Zynq's two high-performance ARM® Cortex™-A9 processors to quickly implement applications that need to deliver deterministic, real-time responsiveness for markets such as automotive, industrial and others with similar requirements. For further information, please refer to Xilinx documentation

AMP demos are based on OpenAMP Framework for Zynq Devices (see UG1186). Open Asymmetric Multi-processing (OpenAMP) is a framework providing the software components needed to enable the development of software applications for asymmetric multi-processing (AMP) systems. The Xilinx implementation of OpenAMP framework provides the following for Zynq-7000 All Programmable (AP) SoC devices:

• The remoteproc , RPMsg , and virtIO components that are used for a Linux master or a bare-metal remote configuration.
• Proxy infrastructure and demos that showcase the ability of a proxy on a master processor running Linux on the ARM processor unit (APU) to handle printf , scanf , open , close , read , and write calls from a bare-metal OS-based remote contexts running on the remote processor unit (RPU).

Build the DESK-XZ7-AN-002

Petalinux build for Kernel

To perform DESK-XZ7-AN-002, you can use a pre-build image or build the components manually with Petalinux. In the latter case the users can build the Petalinux BSP as described here for BORA platform. Some changes must be made to build Petalinux for DESK-XZ7-AN-002. Infact we need to update project-spec/meta-user/recipes-bsp/device-tree/files/system-user.dtsi file as shown below.

diff --git a/project-spec/meta-user/recipes-bsp/device-tree/files/system-user.dtsi b/project-spec/meta-user/recipes-bsp/device-tree/files/system-user.dtsi
index f17536c..d1fc648 100644
--- a/project-spec/meta-user/recipes-bsp/device-tree/files/system-user.dtsi
+++ b/project-spec/meta-user/recipes-bsp/device-tree/files/system-user.dtsi
@@ -1,7 +1,7 @@
 /include/ "system-conf.dtsi"

 // To use openAMP demos uncomment the following line
-// /include/ "openamp.dtsi"
+/include/ "openamp.dtsi"

 //To use eth1 on PL uncomment the following line
 // /include/ "eth1.dtsi"
(END)

After this change you can perform command described here for BORA platform and install artifact bora_openamp_petalinux-image-minimal.wic.bz2 on your uSD.

AMP Demos application build

AMP demos applications can be build as shown below.

AMP Demos application build with Petalinux

In order to buid AMP demos application with Petalinux, we neeed to active the following configuration CONFIG_packagegroup-petalinux-openamp with petalinux-config -c rootf in submenu Petalinux Package Groups. This configuration is already active in the current BSP.

We can check firmware installed on rootfs in /lib/firmware/ directory.

root@bora:~# ls -la /lib/firmware/
total 5452
drwxr-xr-x 2 root root    4096 Mar  9 23:43 .
drwxr-xr-x 8 root root    4096 Mar  9 12:34 ..
-rw-r--r-- 1 root root 2714740 Mar  9 12:34 image_echo_test
-rw-r--r-- 1 root root 2714996 Mar  9 12:34 image_matrix_multiply
-rw-r--r-- 1 root root 3049520 Mar  9 12:34 image_rpc_demo
-rwxr-xr-x 1 root root 3113036 Mar  9 23:43 openamp_echo_project.elf

We can check demo installed on rootfs in /usr/bin/ directory.

root@bora:~# ls -la /usr/bin/echo_test
-rwxr-xr-x 1 root root 9616 Mar  9 12:34 /usr/bin/echo_test
root@bora:~# ls -la /usr/bin/mat_mul_demo
-rwxr-xr-x 1 root root 9616 Mar  9 12:34 /usr/bin/mat_mul_demo
root@bora:~# ls -la /usr/bin/proxy_app
-rwxr-xr-x 1 root root 9636 Mar  9 12:34 /usr/bin/proxy_app
root@bora:~#

Echo demo

For example, if we want execute image_echo_test firmware with echo_test test, we need to perform the following command on target. For more information see UG1186

1. Load the Echo test firmware and RPMsg module:

root@bora:~# dmesg -n 8
root@bora:~# echo image_echo_test > /sys/class/remoteproc/remoteproc0/firmware
root@bora:~# echo start > /sys/class/remoteproc/remoteproc0/state
remoteproc remoteproc0: powering up remoteproc@0
remoteproc remoteproc0: Booting fw image image_echo_test, size 2713244
remoteproc0#vdev0buffer: registered virtio0 (type 7)
remoteproc remoteproc0: remote processor remoteproc@0 is now up
virtio_rpmsg_bus virtio0: rpmsg host is online
virtio_rpmsg_bus virtio0: creating channel rpmsg-openamp-demo-channel addr 0x0

2. Run echo_test

root@bora:~# echo_test

 Echo test start

Master>probe rpmsg_char

 Open rpmsg dev virtio0.rpmsg-openamp-demo-channel.-1.0!
Opening file rpmsg_ctrl0.
checking /sys/class/rpmsg/rpmsg_ctrl0/rpmsg0/name
svc_name: rpmsg-openamp-demo-channel
.

 **************************************

  Echo Test Round 0

 **************************************

 sending payload number 0 of size 9
echo test: sent : 9
 received payload number 0 of size 9

 sending payload number 1 of size 10
echo test: sent : 10
 received payload number 1 of size 10

 sending payload number 2 of size 11
echo test: sent : 11
 received payload number 2 of size 11

 sending payload number 3 of size 12
echo test: sent : 12
 received payload number 3 of size 12

[...]

 sending payload number 469 of size 478
echo test: sent : 478
 received payload number 469 of size 478

 sending payload number 470 of size 479
echo test: sent : 479
 received payload number 470 of size 479

 sending payload number 471 of size 480
echo test: sent : 480
 received payload number 471 of size 480

 **************************************

 Echo Test Round 0 Test Results: Error count = 0

 **************************************
root@bora:~#

3. After you have completed the test, unload the application:

root@bora:~# echo stop > /sys/class/remoteproc/remoteproc0/state

Matrix multiply demo

For example, if we want execute image_matrix_multiply firmware with mat_mul_demo test, we need to perform the following command on target. For more information see UG1186

1. Load the image_matrix_multiply firmware and RPMsg module:

root@bora:~# echo image_matrix_multiply > /sys/class/remoteproc/remoteproc0/firmware
root@bora:~# echo start > /sys/class/remoteproc/remoteproc0/state
[  801.495734] remoteproc remoteproc0: powering up remoteproc@0
[  801.506257] remoteproc remoteproc0: Booting fw image image_matrix_multiply, size 2714996
[  801.649287] virtio_rpmsg_bus virtio0: rpmsg host is online
[  801.653005] virtio_rpmsg_bus virtio0: creating channel rpmsg-openamp-demo-channel addr 0x0
[  801.664077]  remoteproc0#vdev0buffer: registered virtio0 (type 7)
[  801.670986] remoteproc remoteproc0: remote processor remoteproc@0 is now up

2. Run mat_mul_demo

root@bora:~# mat_mul_demo

 Matrix multiplication demo start

Master>probe rpmsg_char
+ lsmod
Module                  Size  Used by
rpmsg_char             16384  0
virtio_rpmsg_bus       20480  0
rpmsg_core             16384  2 rpmsg_char,virtio_rpmsg_bus
zynq_remoteproc        16384  0
uio_pdrv_genirq        16384  0
+ modprobe rpmsg_char

 Open rpmsg dev virtio0.rpmsg-openamp-demo-channel.-1.0!
open /dev/rpmsg_ctrl0
checking /sys/class/rpmsg/rpmsg_ctrl0/rpmsg0/name
svc_name: rpmsg-openamp-demo-channel
.
open /dev/rpmsg0
main:297 matrix_mult(1)

 Master : Linux : Input matrix 0

 2  5  1  0  7  8
 1  7  5  4  2  1
 0  7  0  9  0  1
 6  9  5  3  4  1
 2  3  3  8  3  0
 6  5  5  7  7  4

 Master : Linux : Input matrix 1

 7  1  3  2  5  5
 4  5  5  6  6  7
 9  2  7  4  8  3
 7  0  6  0  0  2
 0  6  7  7  5  7
 1  3  8  4  7  5
0: write rpmsg: 296 bytes
read results

 Master : Linux : Printing results
 51  95  151  119  139  137
 109  61  119  82  104  96
 92  38  97  46  49  72
 145  88  152  118  151  147
 109  41  111  55  67  77
 160  95  201  127  163  163
End of Matrix multiplication demo Round 0

 Quitting application ..
 Matrix multiply application end

3. After you have completed the test, unload the application:

root@bora:~# echo stop > /sys/class/remoteproc/remoteproc0/state
[ 1115.385672] remoteproc remoteproc0: stopped remote processor remoteproc@0

Proxy demo

For example, if we want execute proxy application we need to perform the following command on target. For more information see UG1186

root@bora:~# proxy_app

Master>Loading remote firmware
[ 1210.771013] remoteproc remoteproc0: powering up remoteproc@0
[ 1210.783220] remoteproc remoteproc0: Booting fw image image_rpc_demo, size 3049520
[ 1210.930727] virtio_rpmsg_bus virtio0: rpmsg host is online
[ 1210.937897]  remoteproc0#vdev0buffer: registered virtio0 (type 7)
[ 1210.944037] virtio_rpmsg_bus virtio0: creating channel rpmsg-openamp-demo-channel addr 0x0
[ 1210.953077] remoteproc remoteproc0: remote processor remoteproc@0 is now up

Master>probe rpmsg_char
Opening file rpmsg_ctrl0.
checking /sys/class/rpmsg/rpmsg_ctrl0/rpmsg0/name
svc_name: rpmsg-openamp-demo-channel
.

Master>RPC service started !!

Remote>Baremetal Remote Procedure Call (RPC) Demonstration

Remote>***************************************************

Remote>Rpmsg based retargetting to proxy initialized..

Remote>FileIO demo ..

Remote>Creating a file on master and writing to it..

Remote>Opened file 'remote.file' with fd = 7

Remote>Wrote to fd = 7, size = 45, content = This is a test string being written to file..

Remote>Closed fd = 7

Remote>Reading a file on master and displaying its contents..

Remote>Opened file 'remote.file' with fd = 7
handle_read: 4, 90

Remote>Read from fd = 7, size = 90, printing contents below .. This is a test string being written to file..This is a test string being written to file..

Remote>Closed fd = 7

Remote>Remote firmware using scanf and printf ..

Remote>Scanning user input from master..

Remote>Enter name
Dave
handle_read: 4, 5

Remote>Enter age
25
handle_read: 4, 3

Remote>Enter value for pi
3.1415926535897932384626433832795
handle_read: 4, 34

Remote>User name = 'Dave'

Remote>User age = '25'

Remote>User entered value of pi = '3.141593'

Remote>Repeat demo ? (enter yes or no)
yes
handle_read: 4, 4

Remote>Remote firmware using scanf and printf ..

Remote>Scanning user input from master..

Remote>Enter name
Dave
handle_read: 4, 5

Remote>Enter age
25
handle_read: 4, 3

Remote>Enter value for pi
3.1415926535897932384626433832795
handle_read: 4, 34

Remote>User name = 'Dave'

Remote>User age = '25'

Remote>User entered value of pi = '3.141593'

Remote>Repeat demo ? (enter yes or no)
no
handle_read: 4, 3

Remote>RPC retargetting quitting ...

Remote[ 1234.438610] virtio_rpmsg_bus virtio0: destroying channel rpmsg-openamp-demo-channel addr 0x0
> Firmware's rpmsg-rpc-channel going down!

Master>RPC service exiting !!
[ 1235.575670] remoteproc remoteproc0: stopped remote processor remoteproc@0

AMP Demos application build manually

In order to build Linux Demo Applications we need to perform the following command on host

source /opt/Xilinx/petalinux_2021.2_toolchain/environment-setup-cortexa9t2hf-neon-xilinx-linux-gnueabi
git clone https://github.com/Xilinx/meta-openamp -b rel-v2021.2
cd meta-openamp/recipes-openamp/rpmsg-examples/rpmsg-echo-test/
make

norlando@focalbakery:~/yocto$ source /opt/Xilinx/petalinux_2021.2_toolchain/environment-setup-cortexa9t2hf-neon-xilinx-linux-gnueabi
norlando@focalbakery:~/yocto$ git clone https://github.com/Xilinx/meta-openamp -b rel-v2021.2                                                                                                                                                                                                                                                                                                                                                               Cloning into 'meta-openamp'...
remote: Enumerating objects: 1926, done.
remote: Counting objects: 100% (929/929), done.
remote: Compressing objects: 100% (411/411), done.
remote: Total 1926 (delta 468), reused 858 (delta 410), pack-reused 997
Receiving objects: 100% (1926/1926), 285.07 KiB | 4.01 MiB/s, done.
Resolving deltas: 100% (1029/1029), done.
norlando@focalbakery:~/yocto$ cd meta-openamp/recipes-openamp/rpmsg-examples/rpmsg-echo-test/
norlando@focalbakery:~/yocto/meta-openamp/recipes-openamp/rpmsg-examples/rpmsg-echo-test$ make -j $nproc
arm-xilinx-linux-gnueabi-gcc  -mthumb -mfpu=neon -mfloat-abi=hard -mcpu=cortex-a9 -fstack-protector-strong  -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -Werror=format-security --sysroot=/opt/Xilinx/petalinux_2021.2_toolchain/sysroots/cortexa9t2hf-neon-xilinx-linux-gnueabi -c  -O2 -pipe -g -feliminate-unused-debug-types  -o echo_test.o echo_test.c
echo_test.c: In function 'get_rpmsg_ept_dev_name':
echo_test.c:83:3: warning: ignoring return value of 'fgets' declared with attribute 'warn_unused_result' [-Wunused-result]
   83 |   fgets(svc_name, sizeof(svc_name), fp);
      |   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
echo_test.c: In function 'main':
echo_test.c:163:25: warning: '%s' directive writing up to 255 bytes into a region of size 251 [-Wformat-overflow=]
  163 |    sprintf(fpath, "/dev/%s", ent->d_name);
      |                         ^~
In file included from /opt/Xilinx/petalinux_2021.2_toolchain/sysroots/cortexa9t2hf-neon-xilinx-linux-gnueabi/usr/include/stdio.h:866,
                 from echo_test.c:18:
/opt/Xilinx/petalinux_2021.2_toolchain/sysroots/cortexa9t2hf-neon-xilinx-linux-gnueabi/usr/include/bits/stdio2.h:38:10: note: '__builtin___sprintf_chk' output between 6 and 261 bytes into a destination of size 256
   38 |   return __builtin___sprintf_chk (__s, __USE_FORTIFY_LEVEL - 1,
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
   39 |       __bos (__s), __fmt, __va_arg_pack ());
      |       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
echo_test.c:171:30: warning: '%s' directive writing up to 255 bytes into a region of size 16 [-Wformat-overflow=]
  171 |    sprintf(rpmsg_ctrl_name, "%s", ent->d_name);
      |                              ^~
In file included from /opt/Xilinx/petalinux_2021.2_toolchain/sysroots/cortexa9t2hf-neon-xilinx-linux-gnueabi/usr/include/stdio.h:866,
                 from echo_test.c:18:
/opt/Xilinx/petalinux_2021.2_toolchain/sysroots/cortexa9t2hf-neon-xilinx-linux-gnueabi/usr/include/bits/stdio2.h:38:10: note: '__builtin___sprintf_chk' output between 1 and 256 bytes into a destination of size 16
   38 |   return __builtin___sprintf_chk (__s, __USE_FORTIFY_LEVEL - 1,
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
   39 |       __bos (__s), __fmt, __va_arg_pack ());
      |       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
arm-xilinx-linux-gnueabi-gcc  -mthumb -mfpu=neon -mfloat-abi=hard -mcpu=cortex-a9 -fstack-protector-strong  -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -Werror=format-security --sysroot=/opt/Xilinx/petalinux_2021.2_toolchain/sysroots/cortexa9t2hf-neon-xilinx-linux-gnueabi -Wl,-O1 -Wl,--hash-style=gnu -Wl,--as-needed -Wl,-z,relro,-z,now -o echo_test echo_test.o
norlando@focalbakery:~/yocto/meta-openamp/recipes-openamp/rpmsg-examples/rpmsg-echo-test$ ls -la
total 108
drwxrwxr-x 2 norlando devel  4096 Jan 19 15:10 .
drwxrwxr-x 5 norlando devel  4096 Jan 19 15:09 ..
-rw-rw-r-- 1 norlando devel  1686 Jan 19 15:09 LICENSE
-rw-rw-r-- 1 norlando devel   237 Jan 19 15:09 Makefile
-rwxrwxr-x 1 norlando devel 38748 Jan 19 15:10 echo_test
-rw-rw-r-- 1 norlando devel  8301 Jan 19 15:09 echo_test.c
-rw-rw-r-- 1 norlando devel 39744 Jan 19 15:10 echo_test.o
norlando@focalbakery:~/yocto/meta-openamp/recipes-openamp/rpmsg-examples/rpmsg-echo-test$

After build manually echo-test, we need to transfer binary into target.

Next step is build FreeRTOS Firmware for the openAMP demos, it can be built with Vitis.

source /opt/Xilinx/2021.2/Vitis/2021.2/settings64.sh
vitis

For generate openamp_echo.elf, please perform the following operations from the GUI:

• Click on File->New->Application Project
• Click on Next
• Select tab Create a new platform from hardware (CSA)
• Click on Browse and select the XSA file exported from you Vivado project. For more information about generate xsa file, see here
• Set the Platform name to bora_platform and click Next
• Set the Project Name: openamp_echo (or openamp_matrix_mult or openamp_rpc for the other demos)
• Select the processor ps7_cortexa9_1 and click Next
• Select the operating system freertos10_xilinx and click Next
• Select OpenAMP echo-test (or OpenAMP matrix multiplication Demo or OpenAMP RPC Demo for the other demos) and click Finish
• On Project Explorer tab, open bora_platform and double-click on platform.spr
• Select bora_platform->ps7_cortexa9_1->freertos10_xilinx_ps7_cortexa9_1>Board Support Package
• Click on Modify BSP Settings...
• Under Overview->freertos10_xilinx->openamp: set WITH_PROXY parameter to FALSE for echo and matrix demos, set to TRUE for proxy demo
• Under drivers>ps7_cortexa9_1 append -DUSE_AMP=1 to extra_compiler_flags parameter
• Close settings by clicking OK button
• Rebuild project by right clicking on the project name openamp_echo in Project Explorer and selecting first Clean Project and then Build Project

The FreeRTOS Firmware demo project (ELF file) is then built and available under the Debug directory of the project. Copy the openamp_echo.elf (or openamp_matrix_mult.elf or openamp_rpc.elf for the other demos) into the target root filesystem under the /lib/firmware/ directory

root@bora:~# echo openamp_echo.elf > /sys/class/remoteproc/remoteproc0/firmware
root@bora:~# echo start > /sys/class/remoteproc/remoteproc0/state
root@bora:~# ./echo_test

 Echo test start

Master>probe rpmsg_char

 Open rpmsg dev virtio0.rpmsg-openamp-demo-channel.-1.0!
Opening file rpmsg_ctrl0.
checking /sys/class/rpmsg/rpmsg_ctrl0/rpmsg0/name
svc_name: rpmsg-openamp-demo-channel
.

 **************************************

  Echo Test Round 0

 **************************************

 sending payload number 0 of size 9
echo test: sent : 9
 received payload number 0 of size 9

 sending payload number 1 of size 10
echo test: sent : 10
 received payload number 1 of size 10

 sending payload number 2 of size 11
echo test: sent : 11
 received payload number 2 of size 11

 sending payload number 3 of size 12
echo test: sent : 12
 received payload number 3 of size 12

 sending payload number 4 of size 13
echo test: sent : 13
 received payload number 4 of size 13

 sending payload number 5 of size 14
echo test: sent : 14
 received payload number 5 of size 14

 sending payload number 6 of size 15
echo test: sent : 15
 received payload number 6 of size 15

[...]

 sending payload number 469 of size 478
echo test: sent : 478
 received payload number 469 of size 478

 sending payload number 470 of size 479
echo test: sent : 479
 received payload number 470 of size 479

 sending payload number 471 of size 480
echo test: sent : 480
 received payload number 471 of size 480

 **************************************

 Echo Test Round 0 Test Results: Error count = 0

 **************************************