dave_user, Administrators
5,138
edits
Changes
no edit summary
{{InfoBoxTop}}
{{AppliesToMachineLearning}}
{{AppliesTo Machine Learning TN}}
[[Category:MISC-AN-TN]]
[[Category:MISC-TN]]
|August 2020
|First public release
|-
|1.0.1
|August 2020
|Minor fixes
|}
In essence, the system consists of a high-frame-rate image sensor (*) interfaced to a Linux-based embedded platform also denoted as PP in the rest of the document. The sensor frames a specific area of the production line and sends a constant-rate flow of frames to the PP for further processing, as detailed in the following sections.
The first part is a progressive counter, starting from 000000. In other words, the image whose file name is something like 000000_x.bmp refers to the frame captured at the beginning of the recording window (t<sub>B</sub>). At the other end, the image whose file name starts with the highest counter refers to the frame captured at the end of the recording window (t<sub>A</sub>). This scheme is convenient for automatically processing the frames because it allows to order them very easily.
The second part of the file names is the relative time offset (expressed in seconds) with respect to the "alarm frame." The alarm frame, in turn, is the first frame captured after the detection of an alarm signal. Consequently, the alarm frame is always named as <code>n_+0.000000000.bmp</code>. In other words, this frame is associate associated with t<sub>0</sub>. In the example shown above, the alarm frame's name is <code>002424_+0.000000000.bmp</code>. This rule allows to determine straightforwardly how close a frame is to the alarm event. For instance, see the following image that refers to a screenshot captured on the processing platform itself.
As stated previously, a an H.264 video file is also generatedby combining all the saved frames. This can be convenient for users when they wish to analyze the evolution of the framed scene from a dynamic perspective.
==Real-timeness, frame droppingloss, and alarm detection latency==
One of the major concerns that were faced during the conception of the PoC was related to its real-timeness. In principle, two real-time requirements have to be met to achieve the desired functionalities:
# [REQ1] Regarding the acquisition of frames from the image sensor, such a PoC should guarantee that all frames are captured (i.e. no frame loss), no matter what processing the PP is performing simultaneously (encoding the live stream, serving a web page, etc.). Taking into account the frame rate and the fact that the sensor has no buffering capabilities, this implies that the processing platform has about 3ms to complete the acquisition of each frame.
The top part shows the ideal recording buffer. As described in section [[#Sizing the alarm buffer|''Sizing the alarm buffer'']], the recording is supposed to start 8 seconds (t<sub>B</sub>) before the occurrence of the alarm event (t<sub>0</sub>) and end 3 seconds afterwards (t<sub>A</sub>). As a matter of fact, the recording system takes some time to recognize that the alarm signal is asserted—we refer to this time as ''alarm detection latency''. The fact that this latency is greater than 0 leads to the actual situation illustrated in the bottom part of the picture. Let t<sub>0'</sub> be the actual instant when the PP detects the assertion of the alarm signal. Consequently, the beginning and the end of the recording buffer (t<sub>B'</sub> and t<sub>A'</sub> respectively) are delayed as well. Of course, alarm detection latency should be minimized in order to get the actual recording buffer as close as possible to the ideal one. If the latency is too large, in fact, the recording could miss essential events. For instance, let's assume that the alarm was triggered by a failure occurring at time t<sub>F</sub> as shown in the picture. Because of the alarm detection latency, this instant is before the start of the actual recording buffer (t<sub>F</sub> precedes t<sub>B'</sub>) and thus it is not available for the post-mortem analysis.
In light of these requirements, it is clear that the processing platform should be engineered as a real-time systemto avoid exceeding a maximum latency. Despite these considerations, however, a platform not specifically designed with real-time capabilities was utilized. Rather, it was decided to use a platform providing considerable headroom in terms of processing power and to verify experimentally the fulfillment of requirements. This approach allowed to meet the budget available for the PoC by limiting implementation effort and development time. Basically, two different approaches were utilized for experimental verification.* With regard to frame loss, a mechanism able to detect this anomaly was implemented at the application software level. Should a frame loss occur, this event is logged and reported as shown by the example log below (see "Abnormal frame rate" messages) (*). Incidentally, this mechanism exploits frame timestamps described in [[#Alarm recordings|this section]].
<pre class="board-terminal">
[2020-08-10 10:48:53,674][INFO][main_logger.main] SAVING_ALARM_BUFFER -> STREAM_ONLY [2020-08-10 10:48:53,675][INFO][main_logger.main] STREAM_ONLY -> WAITING_FOR_ALARM [2020-08-10 10:49:22,570][DEBUG][main_logger.main] Alarm signal asserted [2020-08-10 10:49:22,571][INFO][main_logger.main] WAITING_FOR_ALARM -> FILLING_ALARM_BUFFER [2020-08-10 10:49:23,571][DEBUG][main_logger.main] Alarm signal deasserted [2020-08-10 10:49:25,631][INFO][main_logger.alarmbuffersaverthread] Starting AlarmBufferSaverThread ... [2020-08-10 10:49:25,631][INFO][main_logger.main] FILLING_ALARM_BUFFER -> SAVING_ALARM_BUFFER [2020-08-10 10:49:25,678][ERROR][main_logger.main] Abnormal frame rate (42.6 fps) [2020-08-10 10:49:25,716][INFO][main_logger.alarmbuffersaverthread] Deleting oldest alarm ... [2020-08-10 10:49:25,778][ERROR][main_logger.main] Abnormal frame rate (59.6 fps) [2020-08-10 10:49:25,839][ERROR][main_logger.main] Abnormal frame rate (74.5 fps) [2020-08-10 10:49:25,938][INFO][main_logger.alarmbuffersaverthread] Saving to /mnt/alarms/2020-08-10_10.49.22_CEST [2020-08-10 10:50:45,750][ERROR][main_logger.main] Stopping to stream frames ... [2020-08-10 10:50:45,784][ERROR][main_logger.main] It is enough data for buffer ... [2020-08-10 10:50:48,822][ERROR][main_logger.main] Stopping to stream frames ... [2020-08-10 10:50:48,855][ERROR][main_logger.main] It is enough data for buffer ... [2020-08-10 10:51:02,582][DEBUG][main_logger.main] Alarm signal asserted [2020-08-10 10:51:03,581][DEBUG][main_logger.main] Alarm signal deasserted [2020-08-10 10:51:59,825][INFO][main_logger.alarmbuffersaverthread] Ending AlarmBufferSaverThread ... [2020-08-10 10:51:59,844][INFO][main_logger.main] SAVING_ALARM_BUFFER -> STREAM_ONLY [2020-08-10 10:51:59,844][INFO][main_logger.main] STREAM_ONLY -> WAITING_FOR_ALARM
</pre>
* Concerning alarm detection latency, a tailor-made testbed was built as illustrated in the next section.
(*) Incidentally, this mechanism exploits frame timestamps described in [[#Alarm recordings|this section]].
===Alarm detection latency verification===
The following diagram shows how the specific testbed used for this verification looks like.
In essence, the idea consists of shooting a 5-digit 7-segment-display timer with 1ms resolution. The timer cycles repeatedly from 00.000s to 99.999s and also provides a digital signal that is used to emulate the alarm signal. Whenever the timer reaches 50.000s, this signal is raised for 1 second. In other words, the timer triggers an emulated alarm that is synchronized with the visualization of the 50th second on the displays. Referring to the Fig.X TBD2, istants t<sub>B</sub>, t<sub>0</sub>, and t<sub>A</sub> can be easily associated with the time visualized by the timer as follows:
*t<sub>B</sub> = 50.000-8.000 = 42.000s
*t<sub>0</sub> = 50.000s
*The timer displaying 50.000.
Let's analyze one of the recording buffers that were captured this way. It refers to the alarm recorded on Agust 8, 2020 at 12.09.51 CEST. The following images show the frame associated with t<sub>B'</sub>, t<sub>A'</sub>, and t<sub>0'</sub> respectively.
*t<sub>B'</sub> frame
**This frame is named <code>000000_-8.133780480.png</code>
**The timer displays 41.86x(expected 42.000)
*t<sub>A'</sub> frame
**This frame is named <code>003332_+3.046812160.png</code>
**The timer displays 53.04x(expected 53.000)
What about t<sub>0'</sub> frame? It is named correctly (<code>002424_+0.000000000.png</code>), but the timer displays 98.88x, which is not what we expected at all. Why? To find the answer, let's have a look at the subsequent frame:
Timer displays 50.00x. This means that, in the middle of the exposure window associated with the alarm frame, the timer refreshed all the 7-segment displays from something like 49.xyz to 50.abc. Thus, all the digits were affected by a change during the acquisition and this is the reason why they do not appear consistently.
In conclusion, the number of stored frames is correct (3333) and the assessment of these frames—which the aforementioned specific frames was repeated over several recordings—was completed successfully because the discrepancies are in the order of hundreds of milliseconds, therefore much smaller than the size of the recording window. These checks were repeated over several recordings and similar results were found. Nevertheless, it is also worth mentioning that this analysis does not allow us to say that the processing platform behaves like a true real-time system when it comes with regard to acquiring capturing the high-frame-rate flow of frames coming from and processing the image sensoralarm signal. InsteadAs such, we can draw not determine the conclusion upper bound of alarm detection latency, for example. Rather we can say that it the system '''typically''' performs like as in the example detailed here. And this This result is enough to satisfy this the PoC's requirements.
==Future work==
=== Industrialization ===
To turn this PoC into a real product, it should be submitted to an accurate and extensive industrialization process. For instance, the processing platform should be implemented with a true real-time system. Under this assumption, the effective computational power required should be estimated. In turn, this would lead to selecting an embedded platform that is significantly cheaper than the one used for the PoC.
In regard to the application software, it should be completed with full error management and a richer web interface. Additionally, it can not be excluded that parts of it need to be rewritten to achieve desired performances.
===Adding machine learning-based processing===
Speaking of possible future evolutions of this recording/streaming system, many options are available to add even more advanced functionalities. For example, this device could leverage advanced machine learning-based computer vision algorithms to process the high-frame-rate stream. In the world of industrial automation, many issues could be addressed ranging from anomaly detection to predictive maintenance.
For these reasons, [https://www.nxp.com/products/processors-and-microcontrollers/arm-processors/i-mx-applications-processors/i-mx-8-processors/i-mx-8m-plus-arm-cortex-a53-machine-learning-vision-multimedia-and-industrial-iot:IMX8MPLUS NXP i.MX8M Plus] will be one of the key components considered for the possible product industrialization, as DAVE Embedded Systems is part of the beta program associated with this system-on-chip.
== Appendix A ==
In this appendix, the log indicated in section [[#Real-timeness, frame loss, and alarm detection latency|''Real-timeness, frame loss, and alarm detection latency'']] will be further analyzed to provide more details about the application software.
During normal operation, if the alarm detection functionality is enabled, the application waits indefinitely for an alarm to be raised:
<pre class="board-terminal">
[2020-08-10 10:48:53,674][INFO][main_logger.main] SAVING_ALARM_BUFFER -> STREAM_ONLY
[2020-08-10 10:48:53,675][INFO][main_logger.main] STREAM_ONLY -> WAITING_FOR_ALARM
</pre>
When such an event occurs
<pre class="board-terminal">
[2020-08-10 10:49:22,570][DEBUG][main_logger.main] Alarm signal asserted
</pre>
, the application changes state and completes filling the alarm buffer:
<pre class="board-terminal">
[2020-08-10 10:49:22,571][INFO][main_logger.main] WAITING_FOR_ALARM -> FILLING_ALARM_BUFFER
</pre>
As per system requirements, it takes 3 seconds to fill the buffer after the alarm occurrence (from 10:49:22,570 to 10:49:25,631, in this example). Right after filling process is completed, a specific thread is resumed to store the buffer on the disk:<pre class="board-terminal">
[2020-08-10 10:49:23,571][DEBUG][main_logger.main] Alarm signal deasserted
[2020-08-10 10:49:25,631][INFO][main_logger.alarmbuffersaverthread] Starting AlarmBufferSaverThread ...
[2020-08-10 10:49:25,631][INFO][main_logger.main] FILLING_ALARM_BUFFER -> SAVING_ALARM_BUFFER
</pre>
Interestingly, while the PP is saving the recording buffer on the disk, some frames from the image sensor are lost due the non real-time nature of the system. This results in an temporary frame rate drop. As explained previously, this is tolerable.
<pre class="board-terminal">
[2020-08-10 10:49:25,678][ERROR][main_logger.main] Abnormal frame rate (42.6 fps)
[2020-08-10 10:49:25,716][INFO][main_logger.alarmbuffersaverthread] Deleting oldest alarm ...
[2020-08-10 10:49:25,778][ERROR][main_logger.main] Abnormal frame rate (59.6 fps)
[2020-08-10 10:49:25,839][ERROR][main_logger.main] Abnormal frame rate (74.5 fps)
[2020-08-10 10:49:25,938][INFO][main_logger.alarmbuffersaverthread] Saving to /mnt/alarms/2020-08-10_10.49.22_CEST
[2020-08-10 10:50:45,750][ERROR][main_logger.main] Stopping to stream frames ...
[2020-08-10 10:50:45,784][ERROR][main_logger.main] It is enough data for buffer ...
[2020-08-10 10:50:48,822][ERROR][main_logger.main] Stopping to stream frames ...
[2020-08-10 10:50:48,855][ERROR][main_logger.main] It is enough data for buffer ...
</pre>
The following messages refer to a new alarm signal that happened during the writing on to the disk. This alarm is ignored because the system is not designed to process more the one alarm at a time.
<pre class="board-terminal">
[2020-08-10 10:51:02,582][DEBUG][main_logger.main] Alarm signal asserted
[2020-08-10 10:51:03,581][DEBUG][main_logger.main] Alarm signal deasserted
</pre>
Finally, after the completion of storing persistently the buffer, the system is ready to detect and process a new alarm again:
<pre class="board-terminal">
[2020-08-10 10:51:59,825][INFO][main_logger.alarmbuffersaverthread] Ending AlarmBufferSaverThread ...
[2020-08-10 10:51:59,844][INFO][main_logger.main] SAVING_ALARM_BUFFER -> STREAM_ONLY
[2020-08-10 10:51:59,844][INFO][main_logger.main] STREAM_ONLY -> WAITING_FOR_ALARM
</pre>
== Appendix B ==
This section provides some further details regarding the testbed utilized for the alarm detection latency verification.
The following screenshot shows the emulated alarm signal, which is generated by the custom timer. It consists of a 1-second positive pulse.
[[File:MISC-TN-015-timer-alarm-signal.png|thumb|center|600px|Fig. B1: Emulated alarm signal generated by the custom timer]]
Each display is statically driven by a [https://www.onsemi.com/pub/Collateral/MC74HC595-D.PDF 74HC595] shift register in order to avoid multiplexing that would be detrimental for our purpose.
[[File:MISC-TN-015-timer-refresh-alarm.png|thumb|center|600px|Fig. B2: Emulated alarm signal and latch clock when displaying 50.000]]
Fig. B1 shows the emulated alarm signal (magenta) and the signal used to update the 7-segment displays when the timer reaches 50.000 (light blue). This signal is connected to the "latch clock" inputs of the shift registers so that all the displays are updated simultaneously.
Turning on/off LED's is a very fast process, comparable with the propagation time of the shift register itself. Thus, it can be assumed that the displays show 50.000 within a few tens of nanoseconds after the rising edge of the latch clock, that is microseconds before the assertion of the emulated alarm signal. This proves the correctness of the timer implementation.
==Credits==
*Image sourcesIndustrial automation image source:**[https://www.freepik.com/vectors/technology Technology vector created by macrovector - www.freepik.com]
