MISC-TN-015: Proof-of-Concept of an industrial, high-frame-rate video recording/streaming system

Info Box Applies to Machine Learning

History

Version	Date	Notes
1.0.0	August 2020	First public release

Introduction

This Technical Note (TN for short) illustrates a Proof-of-Concept (PoC) that DAVE Embedded Systems made for a customer operating in the industrial automation market. The goal was to build a prototype of a high-frame-rate video recording/streaming system. In a typical scenario, illustrated in the following picture, this device would be used in fast automatic manufacturing lines for two purposes:

• remote monitoring
• detailed off-line "post-mortem" failure analysis.

Typical scenario of utilization

In essence, the system consists of a high-frame-rate image sensor (*) shooting a specific area of the line. The frames captured by the sensors are addressed to an embedded platform for further processing, as detailed in the following sections.

(*) Resolution and frame rate of this stream have to be carefully determined in function of the characteristics of the scene to shoot, first and foremost the speed of moving objects framed by the sensor and its lens. In the case under discussion, the customer specified a resolution of 1280x720, a frame rate of 300 fps, and the use of a global shutter.

Functionalities

Streaming capability is used to monitor the production line remotely. Under normal operation, this is enough for the human operators to get an overview of the line while it is working. For this purpose, a simple low-frame-rate video stream (25 fps or something) over a local area network does the job.

The most interesting functionality is related to recording capability, however. As shown in the previous image, the production line is governed by a Programmable Logic Controller (PLC), which is interfaced to several actuators and sensors. Of course, the line may be subject to different kinds of faults. The most severe—for instance, a major mechanical failure—can lead to the automatic stop of the line. Thanks to the aforementioned sensors, the PLC is aware of such faulty conditions. In the occurrence of these events, it raises an alarm signal directed to the video recording system. Whenever an alarm is detected, the recording system stores on a mass storage device high-frame-rate footage showing what happened right before and right after the alarm event. This footage will be used later on for post-mortem analysis. In other words, automation engineers and maintenance personnel can leverage this fine-grained sequence of frames to analyze in detail the framed scene around the alarm event moment.

Alarm mode

During normal operation, the high-frame-rate stream (indicated by the red flow in the previous picture) is stored in a buffer in RAM memory.

Sizing the alarm buffer

Alarm buffer's size is related to the size of the time window surrounding the alarm event, as depicted in the previous image.

TBD

Let t₀ be the time associated with the occurrence of an alarm. t₀-t_B is the time before the alarm to be recorded. t_A-t₀ is the time after the alarm to be recorded. Consequently, t_B-t_A is the size of the entire window to be recorder. For example, for the the PoC here described:

• t₀-t_B = 8s
• t_A-t₀ = 3s
• t_B-t_A = 11s

Once the size of the time window is knowkn, it is straightforward to determine how much RAM is required for the buffer. In the case under discussion, the size of a frame is approximately 1280x720x8bpp=921600 byte. One second of recording is thus 921600x300≈264 MByte. In conclusion, the buffer has to be at least 264x11=2904 MByte to contain all the required frames.

which, in turn, enters "alarm mode." This mode is used to

The typical scenario in which this device would be used Under normal conditions, this camera streams

   compresses the 60/30fps stream and streams it over Ethernet

Normal mode allows the human operator to remotely supervise the industrial machine where the device is installed

Future work

ML

Credits

• Image sources:
  • Technology vector created by macrovector - www.freepik.com