View source for BELK-TN-001: Real-timeness, system integrity and TrustZone® technology on AMP configuration

{{InfoBoxTop}}
{{Applies To Bora}}
{{Applies To BoraX}}
{{AppliesToBORA_TN}}
{{AppliesToBORA_Xpress_TN}}
{{InfoBoxBottom}}

== History ==
{| class="wikitable" border="1"
!Version
!Date
!BELK version
!Notes
|-
|0.9.0
|September 2015
|[[Bora_Embedded_Linux_Kit_(BELK)#BELK_software_components|3.0.0]]
|Internal draft
|-
|1.0.0
|November 2015
|[[Bora_Embedded_Linux_Kit_(BELK)#BELK_software_components|2.2.0, 3.0.0]]
|First public release
|-
|}

== Introduction ==
Because of widely available Internet connectivity, nowadays security concerns are not longer limited to PCs, servers and workstations but have become common to many embedded systems as well. Several hardware and software technologies have been developed to deal with this kind of challenges. [http://www.arm.com/products/processors/technologies/trustzone/ ARM® TrustZone® technology] is one of these. As stated in <ref name="WP429">Yashu Gosain and Prushothaman Palanichamy, ''Xilinx WP429 - TrustZone Technology Support in Zynq-7000 All Programmable SoCs (v1.0)'', May 20, 2014</ref>, Xilinx Zynq-7000 AP SoC natively supports [http://www.arm.com/products/processors/technologies/trustzone/ TrustZone® technology], since it integrates dual-core ARM® Cortex™-A9 MPCore™ processor. 

Even if this technology has been conceived primarily to address security issues, embedded systems designers can leverage it to implement innovative configurations, satisfying different in nature requirements that typically arise in industrial applications and deep embedded systems{{efn|These kind of requirements are often totally independent of Internet connectivity}}. Two of such requirements are real-timeness and system integrity{{efn|In this context, for the terms '''integrity''' and '''security''' the definitions provided by <ref name="WP461">Ed Hallett, Giulio Corradi, Steven McNeil, ''Xilinx WP461 - Xilinx Reduces Risk and Increases Efficiency for IEC61508 and ISO26262 Certified Safety Applications (v1.0)'', April 9, 2015</ref> are in use:
* '''security''' refers to a system’s immunity to data disclosure or loss as a result of the unlawful electronic penetration of the system’s protections and defenses
* '''integrity''' denotes the certainty that a system cannot be improperly altered.
For the sake of completeness, definition of ''security'' is provided as well:
* '''security''' refers to a system’s immunity to data disclosure or loss as a result of the unlawful 
electronic penetration of the system’s protections and defenses.
}}.

This White Paper describes the TrustZone-based solution that DAVE Embedded Systems has implemented to meet all these requirements on [[:Category:Bora|BORA]] and [[:Category:BoraX|BORAX]] platforms. A technical description of the adopted approach is provided. Also, performance and characterization tests are detailed and considerations about future developments and improvements are included.

It should be remembered that this solution can be considered as a sort of natural evolution on the traditional AMP configuration described in <ref name="AN-BELK-001">DAVE Embedded Systems, [[AN-BELK-001:_Asymmetric_Multiprocessing_(AMP)_on_Bora_–_Linux_FreeRTOS|''AN-BELK-001: Asymmetric Multiprocessing (AMP) on Bora – Linux FreeRTOS'']]</ref>. For this reason, reading of [[AN-BELK-001:_Asymmetric_Multiprocessing_(AMP)_on_Bora_–_Linux_FreeRTOS|this document]] is highly recommended.

==Limitations of traditional configurations==
Xilinx Zynq AP architecture provides unprecedented possibilities in terms of integration. In industrial world applications, this is often leveraged to combine on a single chip the implementation of real-time tasks with generic software applications and functionalities that don't have specific requirements in terms of real-timeness{{efn|Network connectivity is an example of such functionalities.}}. In addition, the flexibility offered by the FPGA - known as Programmable Logic or PL for short - allows system designers to implement in hardware custom IPs to add new interfaces and peripherals or to move processing modules from the software to the hardware realm{{efn|Powerful tools have been introduced in the market recently that facilitate this process significantly. One of these is [http://www.xilinx.com/products/design-tools/software-zone/sdsoc.html SDSoC]. Bora and BoraX are two of the [http://www.xilinx.com/products/design-tools/software-zone/sdsoc.html#boardskits supported hardware platforms].}}.

The following list recaps the typical requirements that such systems must meet. This list has been compiled on the base of real world use cases (specifically medical, transportation, automation and telecom applications):
* <span id="REQ1">REQ1</span> ''Real-time and non real-time tasks integration'': On the same physical component it is required that non real-time and real-time worlds coexist; this allows overall design simplification, BOM reduction and tighter integration.
* <span id="REQ2">REQ2</span> The non real-time world - denoted as W2 in the rest of the document - is based on a well-known operating system such as linux; this operating system is also denote as GPOS (general purpose operating system).
* <span id="REQ3">REQ3</span> The real-time world - denoted as W1 in the rest of the document - is based on a RTOS or a bare metal executable.
* <span id="REQ4">REQ4</span> ''Inter-world communication and data sharing'': the two worlds must have the capability to:
*# communicate via asynchronous mechanisms
*# share data
* <span  id="REQ5">REQ5</span> ''Integrity'': W1 - also known as ''Secure world'' or ''Trust world'' - must guarantee a high reliability level, no matter how the other world behaves; in other words, W1 can not be altered by any kind of actions taken by the code executed in W2 (also called ''Non-secure world'' or ''Non-trust world'').
* <span  id="REQ6">REQ6</span> ''Boot order'': W1 must be the first world to come up.
* <span id="REQ7">REQ7</span> ''Master-slave relationship'': once the system has completed boot process, a master-slave relationship must be established between W1 and W2, in the sense that W1 must have complete control of W2 world (for instance W1 must be able to force the complete reboot of the GPOS).
* <span id="REQ8">REQ8</span> ''L2 cache availability on W2 side'': Basic AMP configurations does not support L2 cache. Generally speaking, this is not an issue on W1 side because
** typical code and data footprints are relatively limited, thus performances are generally not affected  (this is not true instead for W2 world where, in case L2 is not available, overall GPOS performances may decrease noticeably)
** L2 may affect predictability of execution time significantly.

[[AN-BELK-001:_Asymmetric_Multiprocessing_(AMP)_on_Bora_–_Linux_FreeRTOS|Traditional AMP]]<ref name="AN-BELK-001"></ref> configuration satisfies [[#REQ1|REQ1]] through [[#REQ4|REQ4]]. [[#REQ5|REQ5]] through [[#REQ8|REQ8]] are not satisfied instead. About integrity, for example, an application with ''root'' privileges could access memory regions that are supposed to be exclusively accessed by code executed in W1. This may lead to unpredictable behaviors and to potentially catastrophic consequences. This is where TrustZone technology comes to help: it establishes a sort of barrier between the two worlds and prevents W2 code from unauthorized accesses to certain regions of the processor's addressing space.

==TrustZone-based approach==
This section describes in detail the solution implemented by DAVE Embedded Systems to overcome the limitations of basic AMP configuration a to satisfy all the requirements listed in section [[#Limitations of traditional configurations|''Limitations of traditional configurations'']].
===Overview===

The major difference with respect to the traditional AMP configuration is the use of a software monitor, specifically a customized version of TOPPERS SafeG
<ref name="TOPPERS SafeG home (en)">''TOPPERS SafeG home page (English)'', https://www.toppers.jp/en/safeg.html</ref>
<ref name="TOPPERS SafeG home (jp)">''TOPPERS SafeG home page (Japanese)'', https://www.toppers.jp/safeg.html</ref>
<ref name="TOPPERS SafeG">[http://www.wiki.xilinx.com/Multi-OS+Support+%28AMP+%26+Hypervisor%29#Asymmetric%20Multi%20Processing%20%28AMP%29%20Configurations-Open%20Source%20or%20Freely%20Available%20Solutions-TOPPERS%20SafeG%20%28Nagoya%20University%29 ''TOPPERS SafeG (Nagoya University)'']</ref>.

[[File:Safeg-arch-english.png|thumb|center|400px|Nagoya University TOPPERS SafeG architecture]]

As shown in the picture, the monitor can be viewed as a software layer that lies between Trust/Non-trust worlds and underlying hardware. The monitor is responsible for:
* enabling and initializing TrustZone in order to protect regions that must not be accessible by Non-secure world
* setup data structure and exception handlers needed for context switch and Secure Monitor Call (SMC)
* start the trusted OS

Later, once the trusted OS is ready, it will do a specific SMC that will do the context switch that will start the non-trusted OS.

About operating systems, Linux has been chosen for Non-trust world, while [http://www.freertos.org FreeRTOS] has been selected for the Trust world.
At the time of this design, the Linux/FreeRTOS combination has proven to be the most appealing for the majority of applications that this solution addresses. Nevertheless different combinations are possible{{efn|For example TOPPERS project makes use of [http://www.toppers.jp/en/index.html different RTOSes].}}.

About the multi-processing scheme, the two Zynq core are assigned statically to the two world (core0 to Linux, core1 to FreeRTOS). This allows to:
* simplify the whole system implementation
* reduce RTOS latency (because there's never need of ''non-trusted to trusted'' context switch)

From the memory point of view:
* the main memory is statically partitioned (by the monitor) into tree sections:
** a non-trusted private area (protected at MMU-only level from trusted access)
** a trusted private area (protected at TrustZone level by non-trusted access)
** a shared memory area, marked as non-trusted

These choices lead to the configuration depicted [[#L2 cache usage|here]].

===System memory partitioning===
System memory partitioning is shown by the following picture.
[[File:Bora-wp001 02.png|thumb|center|300px|System memory partitioning]]

===Boot process===
The boot process consists of several stages that are detailed in the following list. 
# reset signal is deasserted and core #0's Program Counter is set to reset vector address
# The first piece of code executed by the processor is BootROM. Depending on bootstrap configuration pins, First Stage Boot Loader (FSBL in the rest of the document) image is retrieved from a specific non-volatile memory by BootROM and stored into on-chip memory (OCM).
# FSBL performs basic hardware initializations (including SDRAM subsystem) and retrieves U-Boot bootloader image
# U-Boot
#* completes hardware initializations
#* retrieves the following binary images and store them into SDRAM:
#** monitor
#** trusted code (FreeRTOS image in our case)
#** non-trusted code (linux kernel image and Device Tree Blob in our case).
#* gives monitor the control.
# monitor code
#* initializes TrustZone subsystem 
#* enables both cores, setting up all the data structure required by TrustZone
#* gives trusted code the control of the machine.
# FreeRTOS kernel is initialized and real-time tasks are started. <u>Under the control of the tasks running on top of the RTOS kernel</u>, the non-trusted (NT for short) code is started{{efn|This is done via a Secure Monitor Call (referred as SMC in the rest of the document) that is handled by the monitor.}}.

===Inter-world communication===
Even if it is technically possible to implement a system where W1 and W2 are completely isolated, the majority of real applications need a communication mechanism between the two worlds{{efn|From a different point of view, this is a sort of break in the barrier between the two worlds. As such it poses non-trivial issues in term of integrity and timeliness in the Trust world. This matter will be covered in more detail in following sections.}}. 
Several options are available to implement such a mechanism, each of which having different pros and cons. Exhaustive comparison of all of them is beyond the scope of this paper. Nevertheless some aspects will be briefly discussed and some useful links will be provided to study this notable subject in more depth. 

The following criteria have been taken into account to determine how to select the communication mechanism:
# acceptance into the mainline linux kernel
# possibility to customize the implementation in order to control the degree of isolation between the two worlds.
The first criteria guarantees future maintainability on Linux side. Generally speaking, it is expected that the GPOS needs relatively frequent maintenance activities in order to add new functionalities or fix vulnerabilities and bugs. In contrast, RTOS side is typically more stable and easier to maintain. Therefore, selecting a communication mechanism that is included in mainline Linux kernel is preferable from the point of view of overall system maintainability.

The second criteria is very important because, as discussed in following sections, it is absolutely crucial that the communication channel - <u>that affects directly the degree of isolation between W1 and W2</u> - is very flexible in order to adjust it to application-specific requirements as needed.

Different solutions have been evaluated, including but not limited to 
''OP-TEE''<ref name="OP-TEE">''Open Portable Trusted Execution Environment'', https://www.linaro.org/blog/core-dump/op-tee-open-source-security-mass-market/</ref>, 
''dualoscom''<ref name="Sangorrin's thesis">Daniel Sangorrin Lopez, ''Advanced integration techniques for highly reliable dual-os embedded systems'', Nagoya University (Japan), 27th July 2012, http://ir.nul.nagoya-u.ac.jp/jspui/bitstream/2237/16907/1/k9888.pdf</ref>, 
''RPMsg''<ref name="RPMsg-TI">http://omappedia.org/wiki/Category:RPMsg</ref>, <ref name="RPMsg-linux">https://lwn.net/Articles/464391/</ref> 
and ''OpenAMP''<ref name="OpenAMP">https://github.com/OpenAMP/open-amp</ref>. The choice has fallen on ''RPMsg'' that has been considered the best compromise among the available options. It should be recalled that this choice is reversible, in the sense that if application-specific requirements can not be met by ''RPMsg'', <u>it can be replaced by a different communication scheme</u>.

===L2 cache management===

As specified by [[#REQ8|REQ8]], L2 cache must be able to be enabled (at least) on W2 side. As stated in 
<ref name="XAPP1078"> John McDougall, ''XAPP1078 (v1.0) Simple AMP Running Linux and Bare-Metal System on Both Zynq SoC Processors'', 14th February 2013</ref> and 
<ref name="XAPP1079">John McDougall, ''XAPP1079 (v1.0.1) Simple AMP: Bare-Metal System Running on Both Cortex-A9 Processors'', 24th January 2014</ref>, 
L2 cache - in contrast to L1 - is a shared resource in Zynq implementation. As such, in case both cores have to use it, specific techniques have to be implemented to handle it properly in dual-OS AMP configuration. In principle the approach that has been adopted allows to implement different strategies related to L2 cache management. The actual configuration that has been used to conduct the tests described in [[#Characterization and performance tests|this section]] assigns the whole L2 cache to the W2 world, as depicted in the following picture.

<span id="L2 cache usage"></span>
[[File:Bora-wp001 01 v2-L2cache.png|thumb|center|400px|L2 cache usage]]

This configuration has been chosen for several reasons:
* Linux performance are not affected by this dual-OS solution when accessing it's private memory
* FreeRTOS determinism is granted, because is using only non-shared resources (excluding the SOC L3 bus)
* access to shared memory is uncached or only L1 cached (the latter forces SCU SMP mode usage).
For sake of completeness, it is recalled that it is also possible to enable L2 cache in W1 too, without breaking [[#REQ5|REQ5]], because ARM PL310 L2 cache controller support the TrustZone technology and does not allow the non-trusted OS (W2) to access trusted OS (W1) cached data. However, enabling L2 cache for W1 may improve its computational performance but, in general, reduces real time determinism as well. In case L2 cache unavailability on W1 side is unacceptable, on-chip memory (OCM) can come to help to mitigate this issue.

====Leveraging OCM====
The use of L1 and L2 is strictly related to performance of ARM cores. There's also another precious resource that can increase core performance without affecting determinism, which is the OCM.

In SOCs, OCM is usually tightly coupled to ARM core, providing access performances that are comparable to L2 cache. This allows to leverage it to compensate for the unavailability of L2 memory.

The most common scenario is to:
* restrict OCM access to W1 only (again, this can be done with TrustZone)
* move the most ''latency sensitive'' code - usually vectors and ISRs - inside its memory ranges
* prevent this memory range to be L1-cached (because this usually does not increase the performance significantly but may waste precious L1 memory lines).

==Characterization and performance tests==
Some basics tests have been conducted to characterize the system configured as described above. The figure the tests focus on is the interrupt latency on W1 realm. This value has been measured under different system load conditions to verify if and how the non real-time world may influence the real-time world.

About Linux side, two load conditions have been considered:
* idle
* Google stressapptest (SAT for short) <ref name="SAT">https://code.google.com/p/stressapptest/"</ref> running to stress SDRAM memory and SD I/O.
About RTOS side:
* idle
* memory intensive task; two subcases, in turn, have been considered to evaluate the impact of the L2 cache unavailability.

The RTOS memory task access an array in main memory of two different sizes:
* the smaller is half of L1 size (16KiB)
* the larger is 4 times the L1 size (128KiB)

The main task on the RTOS side is the ''latencystat'' demo provided with [[AN-BELK-001:_Asymmetric_Multiprocessing_(AMP)_on_Bora_–_Linux_FreeRTOS|AN-BELK-001]]<ref name="AN-BELK-001"></ref>, which:
* programs PS TTC timer as freerun, triggering an interrupt on overflow
* inside the overflow ISR the TTC counter is read: this counter reports the number of ticks elapsed between the event (overflow) and the handler itself, in other words the interrupt latency
* after a while the TTC is reprogrammed and interrupt is enabled again, to trigger another event
* those ''latency counters'' are collected into an array
* the Linux-side application, by default after 10 seconds, stops the RTOS task which sends the array data over RPMsg
* the Linux-side application collects the data and display the mininum, maximum and average latency measured

The following table summarize the test results (all timing are given in ''ns'')

{| class="wikitable"
|-
! rowspan="2"| Lantecy !! Linux idle !! colspan="3" | Linux SAT
|-
! RTOS idle !! RTOS idle !! RTOS 16k !! RTOS 128k
|-
| min || align="right"| 287 || align="right"| 287 || align="right"| 287 || align="right"| 1268
|-
| avg || align="right"| 287 || align="right"| 296 || align="right"| 305 || align="right"| 2024
|-
| max || align="right"| 548 || align="right"| 539 || align="right"| 575 || align="right"| 3050
|}

==Conclusions and future work==
The following conclusions can be drawn from the test results:
* Real-timeness of W1 realm is preserved in any condition, since Linux activity on CPU/memory/SD virtually has no influence on RTOS latency.
* Moderate RTOS activity has no impact on latency.
* As expected, in case intensive memory activity is performed on RTOS side, data/instruction cache misses increase significantly resulting in higher latency.
===Future work===
Future work will first focus on an additional feature that has not been included in the requirement list but that is undoubtedly useful in several applications. We are referring to the possibility of performing a complete reboot of the GPOS under the control of the RTOS, while this keeps operating normally. For instance this can be exploited when the RTOS needs to work as software watchdog for W2 activity: in case no activity is detected for a certain period of time, GPOS can be shutdown and rebooted.

Another aspect that should be investigated in more depth refers to the effects of the communication between W1 and W2 on the IRQ latency and the integrity of the real-time world. This matter is strictly related to the degree of isolation between the two worlds. In this work a strong-isolation approach has been adopted, meaning that
*no data is exchanged during the execution of the IRQ latency measurement
*it has been implicitly assumed that data sent from W2 to W1 can not compromise the integrity of the trust domain.
These assumption may be not verified in real applications, however specific techniques can be implemented to manage these situations (see for example <ref name="Sangorrin's thesis"></ref> and <ref name="PreventingInterruptOverload">J. Regehr, U. Duongsaa, ''Preventing Interrupt Overload'', 2nd May 2005, http://www.cs.utah.edu/~regehr/papers/lctes05/regehr-lctes05.pdf</ref>).
-----
{{notelist}}

==References==
{{reflist}}