View source for DESK-XZ7-L-AN-0002: Asymmetric Multiprocessing (AMP) on Bora with OpenAMP

{{InfoBoxTop}}
{{AppliesToBORA_AN}}
{{InfoBoxBottom}}

{{ImportantMessage|text=This application note has been validated using the '''kit version''' in the History table.}}

==History==

{| class="wikitable" border="1"
!Version
!Date
!Development Kit version
|-
| 1.0.0
| Jan 2024
|[[DESK-XZ7-L | DESK-XZ7-L 1.0.1]]
|-
|}

== Introduction ==

Asymmetric Multi Processing (AMP) allows a multiprocessor system to run multiple Operating Systems (OS) that are independent of each other.
In other words, each CPU has its own private memory space, which contains the OS and the applications that are to run on that CPU.
In addition, there can be some shared memory space that is used for multiprocessor communication.
This is contrasted with Symmetric Multiprocessing (SMP), in which one OS runs on multiple CPUs using a public shared memory space.
Thanks to AMP, developers can use open-source Linux and FreeRTOS operating systems and the RPMsg Inter Processor Communication (IPC) framework between the Zynq's two high-performance ARM® Cortex™-A9 processors to quickly implement applications that need to deliver deterministic, real-time responsiveness for markets such as automotive, industrial and others with similar requirements.
For further information, please refer to [http://www.wiki.xilinx.com/Multi-OS+Support+%28AMP+%26+Hypervisor%29 Xilinx documentation]

AMP demos are based on OpenAMP Framework for Zynq Devices (see [https://docs.xilinx.com/v/u/2021.2-English/ug1186-zynq-openamp-gsg UG1186]).
Open Asymmetric Multi-processing (OpenAMP) is a framework providing the software components needed to enable the development of software applications for asymmetric multi-processing (AMP) systems.
The Xilinx implementation of OpenAMP framework provides the following for Zynq-7000 All Programmable (AP) SoC devices:
* The ''remoteproc'' , ''RPMsg'' , and ''virtIO'' components that are used for a Linux master or a bare-metal remote configuration.
* Proxy infrastructure and demos that showcase the ability of a proxy on a master processor running Linux on the ARM processor unit (APU) to handle ''printf'' , ''scanf'' , ''open'' , ''close'' , ''read'' , and ''write'' calls from a bare-metal OS-based remote contexts running on the remote processor unit (RPU).

== Build the DESK-XZ7-AN-002 ==

=== Petalinux build for Kernel ===

To build the components for the DESK-XZ7-AN-002, the users can build the Petalinux BSP as described [https://gitlab.com/DAVEEmbeddedSystems/desk/desk-xz7-l/petalinux here] for ''Bora'' platform.
Some changes must be made to build Petalinux for DESK-XZ7-AN-002. Infact we need to update <code>project-spec/meta-user/recipes-bsp/device-tree/files/system-user.dtsi</code> file as shown below.

<syntaxhighlight lang="diff">
diff --git a/project-spec/meta-user/recipes-bsp/device-tree/files/system-user.dtsi b/project-spec/meta-user/recipes-bsp/device-tree/files/system-user.dtsi
index f17536c..d1fc648 100644
--- a/project-spec/meta-user/recipes-bsp/device-tree/files/system-user.dtsi
+++ b/project-spec/meta-user/recipes-bsp/device-tree/files/system-user.dtsi
@@ -1,7 +1,7 @@
 /include/ "system-conf.dtsi"

 // To use openAMP demos uncomment the following line
-// /include/ "openamp.dtsi"
+/include/ "openamp.dtsi"

 //To use eth1 on PL uncomment the following line
 // /include/ "eth1.dtsi"
(END)
</syntaxhighlight>

After this change you can perform command described [https://gitlab.com/DAVEEmbeddedSystems/desk/desk-xz7-l/petalinux here] for ''Bora'' platform and install artifact on your uSD.

=== AMP Demos application build ===

AMP demos applications can be build as shown below.

==== AMP Demos application build with Petalinux ====

In order to buid AMP demos application with Petalinux, we neeed to active the following configuration <code>CONFIG_packagegroup-petalinux-openamp</code> with <code>petalinux-config -c rootf</code> in submenu <code>Petalinux Package Groups</code>. This configuration is already active in the current BSP.

We can check firmware installed on rootfs in <code>/lib/firmware/</code> directory.

<syntaxhighlight lang="bash">
root@bora:~# ls -la /lib/firmware/
total 5452
drwxr-xr-x 2 root root    4096 Mar  9 23:43 .
drwxr-xr-x 8 root root    4096 Mar  9 12:34 ..
-rw-r--r-- 1 root root 2714740 Mar  9 12:34 image_echo_test
-rw-r--r-- 1 root root 2714996 Mar  9 12:34 image_matrix_multiply
-rw-r--r-- 1 root root 3049520 Mar  9 12:34 image_rpc_demo
-rwxr-xr-x 1 root root 3113036 Mar  9 23:43 openamp_echo_project.elf
</syntaxhighlight>

We can check demo installed on rootfs in <code>/usr/bin/</code> directory.

<syntaxhighlight lang="bash">
root@bora:~# ls -la /usr/bin/echo_test
-rwxr-xr-x 1 root root 9616 Mar  9 12:34 /usr/bin/echo_test
root@bora:~# ls -la /usr/bin/mat_mul_demo
-rwxr-xr-x 1 root root 9616 Mar  9 12:34 /usr/bin/mat_mul_demo
root@bora:~# ls -la /usr/bin/proxy_app
-rwxr-xr-x 1 root root 9636 Mar  9 12:34 /usr/bin/proxy_app
root@bora:~#

</syntaxhighlight>

===== <code>Echo demo</code> =====

For example, if we want execute <code>image_echo_test</code> firmware with <code>echo_test</code> test, we need to perform the following command on target. For more information see [https://docs.xilinx.com/v/u/2021.2-English/ug1186-zynq-openamp-gsg UG1186]

1. Load the Echo test firmware and RPMsg module:

<syntaxhighlight lang="bash">
root@bora:~# dmesg -n 8
root@bora:~# echo image_echo_test > /sys/class/remoteproc/remoteproc0/firmware
root@bora:~# echo start > /sys/class/remoteproc/remoteproc0/state
remoteproc remoteproc0: powering up remoteproc@0
remoteproc remoteproc0: Booting fw image image_echo_test, size 2713244
remoteproc0#vdev0buffer: registered virtio0 (type 7)
remoteproc remoteproc0: remote processor remoteproc@0 is now up
virtio_rpmsg_bus virtio0: rpmsg host is online
virtio_rpmsg_bus virtio0: creating channel rpmsg-openamp-demo-channel addr 0x0
</syntaxhighlight>

2. Run <code>echo_test</code>

<syntaxhighlight lang="bash">
root@bora:~# echo_test

 Echo test start

Master>probe rpmsg_char

 Open rpmsg dev virtio0.rpmsg-openamp-demo-channel.-1.0!
Opening file rpmsg_ctrl0.
checking /sys/class/rpmsg/rpmsg_ctrl0/rpmsg0/name
svc_name: rpmsg-openamp-demo-channel
.

 **************************************

  Echo Test Round 0

 **************************************

 sending payload number 0 of size 9
echo test: sent : 9
 received payload number 0 of size 9

 sending payload number 1 of size 10
echo test: sent : 10
 received payload number 1 of size 10

 sending payload number 2 of size 11
echo test: sent : 11
 received payload number 2 of size 11

 sending payload number 3 of size 12
echo test: sent : 12
 received payload number 3 of size 12

[...]

 sending payload number 469 of size 478
echo test: sent : 478
 received payload number 469 of size 478

 sending payload number 470 of size 479
echo test: sent : 479
 received payload number 470 of size 479

 sending payload number 471 of size 480
echo test: sent : 480
 received payload number 471 of size 480

 **************************************

 Echo Test Round 0 Test Results: Error count = 0

 **************************************
root@bora:~#

</syntaxhighlight>

3. After you have completed the test, unload the application:

<syntaxhighlight lang="bash">
root@bora:~# echo stop > /sys/class/remoteproc/remoteproc0/state
</syntaxhighlight>

===== <code>Matrix multiply demo</code> =====

For example, if we want execute <code>image_matrix_multiply</code> firmware with <code>mat_mul_demo</code> test, we need to perform the following command on target. For more information see [https://docs.xilinx.com/v/u/2021.2-English/ug1186-zynq-openamp-gsg UG1186]

1. Load the <code>image_matrix_multiply</code> firmware and RPMsg module:

<syntaxhighlight lang="bash">
root@bora:~# echo image_matrix_multiply > /sys/class/remoteproc/remoteproc0/firmware
root@bora:~# echo start > /sys/class/remoteproc/remoteproc0/state
[  801.495734] remoteproc remoteproc0: powering up remoteproc@0
[  801.506257] remoteproc remoteproc0: Booting fw image image_matrix_multiply, size 2714996
[  801.649287] virtio_rpmsg_bus virtio0: rpmsg host is online
[  801.653005] virtio_rpmsg_bus virtio0: creating channel rpmsg-openamp-demo-channel addr 0x0
[  801.664077]  remoteproc0#vdev0buffer: registered virtio0 (type 7)
[  801.670986] remoteproc remoteproc0: remote processor remoteproc@0 is now up
</syntaxhighlight>

2. Run <code>echo_test</code>

<syntaxhighlight lang="bash">
root@bora:~# mat_mul_demo

 Matrix multiplication demo start

Master>probe rpmsg_char
+ lsmod
Module                  Size  Used by
rpmsg_char             16384  0
virtio_rpmsg_bus       20480  0
rpmsg_core             16384  2 rpmsg_char,virtio_rpmsg_bus
zynq_remoteproc        16384  0
uio_pdrv_genirq        16384  0
+ modprobe rpmsg_char

 Open rpmsg dev virtio0.rpmsg-openamp-demo-channel.-1.0!
open /dev/rpmsg_ctrl0
checking /sys/class/rpmsg/rpmsg_ctrl0/rpmsg0/name
svc_name: rpmsg-openamp-demo-channel
.
open /dev/rpmsg0
main:297 matrix_mult(1)

 Master : Linux : Input matrix 0

 2  5  1  0  7  8
 1  7  5  4  2  1
 0  7  0  9  0  1
 6  9  5  3  4  1
 2  3  3  8  3  0
 6  5  5  7  7  4

 Master : Linux : Input matrix 1

 7  1  3  2  5  5
 4  5  5  6  6  7
 9  2  7  4  8  3
 7  0  6  0  0  2
 0  6  7  7  5  7
 1  3  8  4  7  5
0: write rpmsg: 296 bytes
read results

 Master : Linux : Printing results
 51  95  151  119  139  137
 109  61  119  82  104  96
 92  38  97  46  49  72
 145  88  152  118  151  147
 109  41  111  55  67  77
 160  95  201  127  163  163
End of Matrix multiplication demo Round 0

 Quitting application ..
 Matrix multiply application end
</syntaxhighlight>

3. After you have completed the test, unload the application:

<syntaxhighlight lang="bash">
root@bora:~# echo stop > /sys/class/remoteproc/remoteproc0/state
[ 1115.385672] remoteproc remoteproc0: stopped remote processor remoteproc@0
</syntaxhighlight>

===== Proxy demo =====

For example, if we want execute proxy application we need to perform the following command on target. For more information see [https://docs.xilinx.com/v/u/2021.2-English/ug1186-zynq-openamp-gsg UG1186]

<pre class="mw-collapsible mw-collapsed workstation-terminal">
root@bora:~# proxy_app

Master>Loading remote firmware
[ 1210.771013] remoteproc remoteproc0: powering up remoteproc@0
[ 1210.783220] remoteproc remoteproc0: Booting fw image image_rpc_demo, size 3049520
[ 1210.930727] virtio_rpmsg_bus virtio0: rpmsg host is online
[ 1210.937897]  remoteproc0#vdev0buffer: registered virtio0 (type 7)
[ 1210.944037] virtio_rpmsg_bus virtio0: creating channel rpmsg-openamp-demo-channel addr 0x0
[ 1210.953077] remoteproc remoteproc0: remote processor remoteproc@0 is now up

Master>probe rpmsg_char
Opening file rpmsg_ctrl0.
checking /sys/class/rpmsg/rpmsg_ctrl0/rpmsg0/name
svc_name: rpmsg-openamp-demo-channel
.

Master>RPC service started !!

Remote>Baremetal Remote Procedure Call (RPC) Demonstration

Remote>***************************************************

Remote>Rpmsg based retargetting to proxy initialized..

Remote>FileIO demo ..

Remote>Creating a file on master and writing to it..

Remote>Opened file 'remote.file' with fd = 7

Remote>Wrote to fd = 7, size = 45, content = This is a test string being written to file..

Remote>Closed fd = 7

Remote>Reading a file on master and displaying its contents..

Remote>Opened file 'remote.file' with fd = 7
handle_read: 4, 90

Remote>Read from fd = 7, size = 90, printing contents below .. This is a test string being written to file..This is a test string being written to file..

Remote>Closed fd = 7

Remote>Remote firmware using scanf and printf ..

Remote>Scanning user input from master..

Remote>Enter name
Dave
handle_read: 4, 5

Remote>Enter age
25
handle_read: 4, 3

Remote>Enter value for pi
3.1415926535897932384626433832795
handle_read: 4, 34

Remote>User name = 'Dave'

Remote>User age = '25'

Remote>User entered value of pi = '3.141593'

Remote>Repeat demo ? (enter yes or no)
yes
handle_read: 4, 4

Remote>Remote firmware using scanf and printf ..

Remote>Scanning user input from master..

Remote>Enter name
Dave
handle_read: 4, 5

Remote>Enter age
25
handle_read: 4, 3

Remote>Enter value for pi
3.1415926535897932384626433832795
handle_read: 4, 34

Remote>User name = 'Dave'

Remote>User age = '25'

Remote>User entered value of pi = '3.141593'

Remote>Repeat demo ? (enter yes or no)
no
handle_read: 4, 3

Remote>RPC retargetting quitting ...

Remote[ 1234.438610] virtio_rpmsg_bus virtio0: destroying channel rpmsg-openamp-demo-channel addr 0x0
> Firmware's rpmsg-rpc-channel going down!

Master>RPC service exiting !!
[ 1235.575670] remoteproc remoteproc0: stopped remote processor remoteproc@0
</pre>

==== AMP Demos application build manually ====

In order to build ''Linux Demo Applications'' we need to perform the following command on host

<syntaxhighlight lang="bash">
source /opt/Xilinx/petalinux_2021.2_toolchain/environment-setup-cortexa9t2hf-neon-xilinx-linux-gnueabi
git clone https://github.com/Xilinx/meta-openamp -b rel-v2021.2
cd meta-openamp/recipes-openamp/rpmsg-examples/rpmsg-echo-test/
make
</syntaxhighlight>


<pre class="mw-collapsible mw-collapsed workstation-terminal">
norlando@focalbakery:~/yocto$ source /opt/Xilinx/petalinux_2021.2_toolchain/environment-setup-cortexa9t2hf-neon-xilinx-linux-gnueabi
norlando@focalbakery:~/yocto$ git clone https://github.com/Xilinx/meta-openamp -b rel-v2021.2                                                                                                                                                                                                                                                                                                                                                               Cloning into 'meta-openamp'...
remote: Enumerating objects: 1926, done.
remote: Counting objects: 100% (929/929), done.
remote: Compressing objects: 100% (411/411), done.
remote: Total 1926 (delta 468), reused 858 (delta 410), pack-reused 997
Receiving objects: 100% (1926/1926), 285.07 KiB | 4.01 MiB/s, done.
Resolving deltas: 100% (1029/1029), done.
norlando@focalbakery:~/yocto$ cd meta-openamp/recipes-openamp/rpmsg-examples/rpmsg-echo-test/
norlando@focalbakery:~/yocto/meta-openamp/recipes-openamp/rpmsg-examples/rpmsg-echo-test$ make -j $nproc
arm-xilinx-linux-gnueabi-gcc  -mthumb -mfpu=neon -mfloat-abi=hard -mcpu=cortex-a9 -fstack-protector-strong  -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -Werror=format-security --sysroot=/opt/Xilinx/petalinux_2021.2_toolchain/sysroots/cortexa9t2hf-neon-xilinx-linux-gnueabi -c  -O2 -pipe -g -feliminate-unused-debug-types  -o echo_test.o echo_test.c
echo_test.c: In function 'get_rpmsg_ept_dev_name':
echo_test.c:83:3: warning: ignoring return value of 'fgets' declared with attribute 'warn_unused_result' [-Wunused-result]
   83 |   fgets(svc_name, sizeof(svc_name), fp);
      |   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
echo_test.c: In function 'main':
echo_test.c:163:25: warning: '%s' directive writing up to 255 bytes into a region of size 251 [-Wformat-overflow=]
  163 |    sprintf(fpath, "/dev/%s", ent->d_name);
      |                         ^~
In file included from /opt/Xilinx/petalinux_2021.2_toolchain/sysroots/cortexa9t2hf-neon-xilinx-linux-gnueabi/usr/include/stdio.h:866,
                 from echo_test.c:18:
/opt/Xilinx/petalinux_2021.2_toolchain/sysroots/cortexa9t2hf-neon-xilinx-linux-gnueabi/usr/include/bits/stdio2.h:38:10: note: '__builtin___sprintf_chk' output between 6 and 261 bytes into a destination of size 256
   38 |   return __builtin___sprintf_chk (__s, __USE_FORTIFY_LEVEL - 1,
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
   39 |       __bos (__s), __fmt, __va_arg_pack ());
      |       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
echo_test.c:171:30: warning: '%s' directive writing up to 255 bytes into a region of size 16 [-Wformat-overflow=]
  171 |    sprintf(rpmsg_ctrl_name, "%s", ent->d_name);
      |                              ^~
In file included from /opt/Xilinx/petalinux_2021.2_toolchain/sysroots/cortexa9t2hf-neon-xilinx-linux-gnueabi/usr/include/stdio.h:866,
                 from echo_test.c:18:
/opt/Xilinx/petalinux_2021.2_toolchain/sysroots/cortexa9t2hf-neon-xilinx-linux-gnueabi/usr/include/bits/stdio2.h:38:10: note: '__builtin___sprintf_chk' output between 1 and 256 bytes into a destination of size 16
   38 |   return __builtin___sprintf_chk (__s, __USE_FORTIFY_LEVEL - 1,
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
   39 |       __bos (__s), __fmt, __va_arg_pack ());
      |       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
arm-xilinx-linux-gnueabi-gcc  -mthumb -mfpu=neon -mfloat-abi=hard -mcpu=cortex-a9 -fstack-protector-strong  -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -Werror=format-security --sysroot=/opt/Xilinx/petalinux_2021.2_toolchain/sysroots/cortexa9t2hf-neon-xilinx-linux-gnueabi -Wl,-O1 -Wl,--hash-style=gnu -Wl,--as-needed -Wl,-z,relro,-z,now -o echo_test echo_test.o
norlando@focalbakery:~/yocto/meta-openamp/recipes-openamp/rpmsg-examples/rpmsg-echo-test$ ls -la
total 108
drwxrwxr-x 2 norlando devel  4096 Jan 19 15:10 .
drwxrwxr-x 5 norlando devel  4096 Jan 19 15:09 ..
-rw-rw-r-- 1 norlando devel  1686 Jan 19 15:09 LICENSE
-rw-rw-r-- 1 norlando devel   237 Jan 19 15:09 Makefile
-rwxrwxr-x 1 norlando devel 38748 Jan 19 15:10 echo_test
-rw-rw-r-- 1 norlando devel  8301 Jan 19 15:09 echo_test.c
-rw-rw-r-- 1 norlando devel 39744 Jan 19 15:10 echo_test.o
norlando@focalbakery:~/yocto/meta-openamp/recipes-openamp/rpmsg-examples/rpmsg-echo-test$
</pre>

After build manually <code>echo-test</code>, we need to transfer binary into target.

Next step is build FreeRTOS Firmware for the openAMP demos, it can be built with Vitis.

<syntaxhighlight lang="bash">
source /opt/Xilinx/2021.2/Vitis/2021.2/settings64.sh
vitis
</syntaxhighlight>

For generate <code>openamp_echo.elf</code>, please perform the following operations from the GUI:

* Click on <code>File->New->Application Project</code>
* Click on <code>Next</code>
* Select tab <code>Create a new platform from hardware (CSA)</code>
* Click on <code>Browse</code> and select the <code>XSA</code> file exported from you Vivado project. For more information about generate <code>xsa</code> file, see [https://gitlab.com/DAVEEmbeddedSystems/desk/desk-xz7-l/petalinux here]
* Set the <code>Platform name</code> to <code>bora_platform</code> and click <code>Next</code>
* Set the <code>Project Name</code>: <code>openamp_echo</code> (or <code>openamp_matrix_mult</code> or <code>openamp_rpc</code> for the other demos)
* Select the processor <code>ps7_cortexa9_1</code> and click <code>Next</code>
* Select the operating system <code>freertos10_xilinx</code> and click <code>Next</code>
* Select <code>OpenAMP echo-test</code> (or <code>OpenAMP matrix multiplication Demo</code> or <code>OpenAMP RPC Demo</code> for the other demos) and click <code>Finish</code>
* On <code>Project Explorer</code> tab, open <code>bora_platform</code> and double-click on <code>platform.spr</code>
* Select <code>bora_platform->ps7_cortexa9_1->freertos10_xilinx_ps7_cortexa9_1>Board Support Package</code>
* Click on <code>Modify BSP Settings...</code>
* Under <code>Overview->freertos10_xilinx->openamp</code>: set <code>WITH_PROXY</code> parameter to <code>FALSE</code> for <code>echo</code> and <code>matrix</code> demos, set to <code>TRUE</code> for <code>proxy</code> demo
* Under <code>drivers>ps7_cortexa9_1</code> append <code> -DUSE_AMP=1</code> to <code>extra_compiler_flags</code> parameter
* Close settings by clicking <code>OK</code> button
* Rebuild project by right clicking on the project name <code>openamp_echo</code> in <code>Project Explorer</code> and selecting first <code>Clean Project</code> and then <code>Build Project</code>

The FreeRTOS Firmware demo project (ELF file) is then built and available under the <code>Debug</code> directory of the project. Copy the <code>openamp_echo.elf</code> (or <code>openamp_matrix_mult.elf</code> or <code>openamp_rpc.elf</code> for the other demos) into the target root filesystem under the <code>/lib/firmware/</code> directory


<syntaxhighlight lang="bash">
root@bora:~# echo openamp_echo.elf > /sys/class/remoteproc/remoteproc0/firmware
root@bora:~# echo start > /sys/class/remoteproc/remoteproc0/state
root@bora:~# ./echo_test

 Echo test start

Master>probe rpmsg_char

 Open rpmsg dev virtio0.rpmsg-openamp-demo-channel.-1.0!
Opening file rpmsg_ctrl0.
checking /sys/class/rpmsg/rpmsg_ctrl0/rpmsg0/name
svc_name: rpmsg-openamp-demo-channel
.

 **************************************

  Echo Test Round 0

 **************************************

 sending payload number 0 of size 9
echo test: sent : 9
 received payload number 0 of size 9

 sending payload number 1 of size 10
echo test: sent : 10
 received payload number 1 of size 10

 sending payload number 2 of size 11
echo test: sent : 11
 received payload number 2 of size 11

 sending payload number 3 of size 12
echo test: sent : 12
 received payload number 3 of size 12

 sending payload number 4 of size 13
echo test: sent : 13
 received payload number 4 of size 13

 sending payload number 5 of size 14
echo test: sent : 14
 received payload number 5 of size 14

 sending payload number 6 of size 15
echo test: sent : 15
 received payload number 6 of size 15

[...]

 sending payload number 469 of size 478
echo test: sent : 478
 received payload number 469 of size 478

 sending payload number 470 of size 479
echo test: sent : 479
 received payload number 470 of size 479

 sending payload number 471 of size 480
echo test: sent : 480
 received payload number 471 of size 480

 **************************************

 Echo Test Round 0 Test Results: Error count = 0

 **************************************
</syntaxhighlight>