DESK-MX9-L-TN-0001: Using gstreamer on AURA
| HOME | SOMs | SBCs | ToloMEO Embedded Assistant |  GET A QUOTE
|
 ONLINE HELPDESK
|
| Roadmap | IoT Services | ML/AI services | Embedded Design Services |
Info Box
|
 This technical note has been validated using the kit version in the History table. |
| History | |||
|---|---|---|---|
| Issue Date | Notes | ||
| DESK-MX9-L 5.1.0 | |||
| 2024/09/18 | PXP plugin example | ||
Contents
Introduction[edit | edit source]
DESK-MX9-L Yocto build target dave-image-devel includes the build of the overall package group for gstreamer:
packagegroup-fsl-gstreamer1.0 \ packagegroup-fsl-gstreamer1.0-full
This allows the possibility of testing many gstreamer features directly in the target.
Despite this Yocto build, some packages are not available due to license restrictions (like libav for the ffmpeg component). For this reason, a special build is required enabling the Commercial license for this testing purposes. Building ffmpeg (a core component for video decoding) reports in fact the following error:
ffmpeg was skipped: because it has a restricted license 'commercial'. Which is not listed in LICENSE_FLAGS_ACCEPTED
this can be overcome by adding the following line
LICENSE_FLAGS_ACCEPTED="commercial"
to the conf/local.conf file in the build directory.
gstreamer version[edit | edit source]
The gstreamer version included in the DESK-MX9-L release desk-mx9-l-5.1.0 is:
root@desk-mx93-rev1:~# gst-launch-1.0 --version gst-launch-1.0 version 1.22.5 GStreamer 1.22.5 Unknown package origin root@desk-mx93-rev1:~#
gst-inspect[edit | edit source]
gst-inspect is a tool that allows to see a lot of information on available GStreamer plugins.
For example, just grepping about the sink plugins, it is possible to see the list of available plugins related to the sink functionality:
root@desk-mx93-rev1:~# gst-inspect-1.0 | grep sink alsa: alsasink: Audio sink (ALSA) app: appsink: AppSink autodetect: autoaudiosink: Auto audio sink autodetect: autovideosink: Auto video sink bluez: a2dpsink: Bluetooth A2DP sink bluez: avdtpsink: Bluetooth AVDTP sink coreelements: fakesink: Fake Sink coreelements: fdsink: Filedescriptor Sink coreelements: filesink: File Sink curl: curlfilesink: Curl file sink curl: curlftpsink: Curl ftp sink curl: curlhttpsink: Curl http sink curl: curlsmtpsink: Curl smtp sink dash: dashsink: DASH Sink debug: testsink: Test plugin debugutilsbad: checksumsink: Checksum sink debugutilsbad: fakeaudiosink: Fake Audio Sink debugutilsbad: fakevideosink: Fake Video Sink debugutilsbad: fpsdisplaysink: Measure and show framerate on videosink debugutilsbad: videocodectestsink: Video CODEC Test Sink decklink: decklinkaudiosink: Decklink Audio Sink decklink: decklinkvideosink: Decklink Video Sink edge: edgesink: EdgeSink fbdevsink: fbdevsink: fbdev video sink gdkpixbuf: gdkpixbufsink: GdkPixbuf sink gio: giosink: GIO sink gio: giostreamsink: GIO stream sink hls: hlssink: HTTP Live Streaming sink hls: hlssink2: HTTP Live Streaming sink inter: interaudiosink: Internal audio sink inter: intersubsink: Internal subtitle sink inter: intervideosink: Internal video sink ipcpipeline: ipcpipelinesink: Inter-process Pipeline Sink multifile: multifilesink: Multi-File Sink multifile: splitmuxsink: Split Muxing Bin nnstreamer: tensor_query_serversink: TensorQueryServerSink nnstreamer: tensor_reposink: TensorRepoSink nnstreamer: tensor_sink: TensorSink opengl: glimagesink: GL Sink Bin opengl: glimagesinkelement: OpenGL video sink opengl: glsinkbin: GL Sink Bin ossaudio: osssink: Audio Sink (OSS) playback: playsink: Player Sink proxy: proxysink: Proxy Sink pulseaudio: pulsesink: PulseAudio Audio Sink rist: ristsink: RIST Sink rtmp2: rtmp2sink: RTMP sink element rtpmanagerbad: rtpsink: RTP Sink element rtspclientsink: rtspclientsink: RTSP RECORD client shm: shmsink: Shared Memory Sink soup: souphttpclientsink: HTTP client sink tcp: multifdsink: Multi filedescriptor sink tcp: multisocketsink: Multi socket sink tcp: tcpclientsink: TCP client sink tcp: tcpserversink: TCP server sink udp: dynudpsink: UDP packet sender udp: multiudpsink: UDP packet sender udp: udpsink: UDP packet sender video4linux2: v4l2sink: Video (video4linux2) Sink waylandsink: waylandsink: wayland video sink root@desk-mx93-rev1:~#
Among these, the main used plugins are:
fbdevsink: for playing the decoded stream on the display directly on fbdevwaylandsink: for playing the decoded stream on the display using the Wayland compositor
libav[edit | edit source]
For the libav libraries, it is possible to check - as an example - the presence of the H.264 decoder:
root@desk-mx93-rev1:~# gst-inspect-1.0 | grep libav | grep 264 libav: avdec_h264: libav H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10 decoder libav: avmux_ipod: libav iPod H.264 MP4 (MPEG-4 Part 14) muxer root@desk-mx93-rev1:~#
Video test pattern[edit | edit source]
A first test, very useful to debug the video output - for example during carrier hardware validation - is the videotestsrc plugin.
It is possible to display a color bar test signal directly on fbdev:
- stop the weston service
systemctl stop weston
- video test at full display resolution
gst-launch-1.0 videotestsrc ! video/x-raw,width=800,height=480 ! fbdevsink
or on top of Wayland:
gst-launch-1.0 videotestsrc ! waylandsink
Video/audio decoding[edit | edit source]
- file: BigBuckBunny_853x480p_h264.mov
- video: H.264 MPEG-4 AVC (part 10) @ 853x480 resolution
- fps: 24
Gstreamer pipeline:
gst-launch-1.0 filesrc location=/home/root/BigBuckBunny_853x480p_h264.mov ! decodebin ! videoconvert ! waylandsink
root@desk-mx93-rev1:~# gst-launch-1.0 filesrc location=/home/root/BigBuckBunny_853x480p_h264.mov ! decodebin ! videoconvert ! waylandsink
Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
====== AIUR: 4.8.2 build on Oct 10 2023 02:31:27. ======
Core: MPEG4PARSER_06.21.06 build on Oct 18 2023 10:08:29
file: /usr/lib/imx-mm/parser/lib_mp4_parser_arm_elinux.so.3.2
------------------------
Track 00 [video_0] Enabled
Duration: 0:09:56.458333000
Language: und
Mime:
video/x-h264, parsed=(boolean)true, alignment=(string)au, stream-format=(string)avc, width=(int)853, height=(int)480, framerate=(fraction)24/1, codec_data=(buffer)014d401effe10015274d401ea9181b07bcde00d4040406db0ad7bdf01001000428de09c8
------------------------
Redistribute latency...
Redistribute latency...
Track 01 [subtitle]: Disabled
Codec: 0, SubCodec: 0
------------------------
------------------------
Track 02 [audio_0] Enabled
Duration: 0:09:56.480000000
Language: und
Mime:
audio/mpeg, mpegversion=(int)4, channels=(int)6, rate=(int)48000, bitrate=(int)448000, stream-format=(string)raw, codec_data=(buffer)11b0
------------------------
====== BEEP: 4.8.2 build on Oct 10 2023 02:31:27. ======
Core: AAC decoder Wrapper build on Sep 8 2023 20:17:24
file: /usr/lib/imx-mm/audio-codec/wrap/lib_aacd_wrap_arm_elinux.so.3
CODEC: BLN_MAD-MMCODECS_AACD_ARM_03.09.00_ARMV8 build on Sep 8 2023 17:16:00.
Redistribute latency...
Redistribute latency...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
Redistribute latency...
New clock: GstSystemClock
0:00:02.5 / 0:09:56.4 (0.4 %)
Using the fpsdisplaysink plugin, i.e. fpsdisplaysink: Measure and show framerate on videosink it is possible to measure the CPU performance about decoding and rendering a video stream:
gst-launch-1.0 filesrc location=/home/root/BigBuckBunny_853x480p_h264.mov ! decodebin ! videoconvert ! fpsdisplaysink
The related CPU load is roughly around 30%
Full HD (1920x1080) video decoding[edit | edit source]
- file: big_buck_bunny_1080p_h264.mov
- video: H.264 MPEG-4 AVC (part 10) @ 1920x1080 resolution
- fps: 24
Full HD video decoding and rendering without a VPU or a GPU - as the i.MX93 SoC - is a challenging task for a CPU software-only processor. This means that video conversion - for the proper rendering on the output display - is quite a hard effort and does not work properly on this kind of CPU.
Using the following movie example:
root@desk-mx93-rev1:~# wget -k https://download.blender.org/peach/bigbuckbunny_movies/big_buck_bunny_1080p_h264.mov --2024-09-18 10:46:06-- https://download.blender.org/peach/bigbuckbunny_movies/big_buck_bunny_1080p_h264.mov Resolving download.blender.org... 104.22.65.163, 172.67.14.163, 104.22.64.163, ... Connecting to download.blender.org|104.22.65.163|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 725106140 (692M) [video/quicktime] Saving to: 'big_buck_bunny_1080p_h264.mov' big_buck_bunny_1080p_h264.mov 100%[================================================================>] 691.51M 11.8MB/s in 68s 2024-09-18 10:47:14 (10.1 MB/s) - 'big_buck_bunny_1080p_h264.mov' saved [725106140/725106140] Converted links in 0 files in 0 seconds. root@desk-mx93-rev1:~#
with a standard gstreamer pipeline:
gst-launch-1.0 filesrc location=/home/root/big_buck_bunny_1080p_h264.mov ! decodebin ! videoconvert ! fpsdisplaysink
results in many dropped frames: see the results here below:
PXP processor[edit | edit source]
The PXP processor can be used - in the i.MX93 - for composition, scaling, rotation or color conversion operations reducing a lot the CPU load.
PXP gstreamer plug-ins can be checked - again - using gst-inspect:
root@desk-mx93-rev1:~# gst-inspect-1.0 | grep pxp imxcompositor: imxcompositor_pxp: IMX pxp Video Compositor imxvideoconvert: imxvideoconvert_pxp: IMX pxp Video Converter root@desk-mx93-rev1:~#
which shows the presence of the Video converter. Then, the pipeline will be the following one:
gst-launch-1.0 filesrc location=/home/root/big_buck_bunny_1080p_h264.mov ! decodebin ! video/x-raw,width=1920,height=1080 ! imxvideoconvert_pxp ! video/x-raw,width=800,height=480 ! fpsdisplaysink
In this way, the video conversion will be performed by the PXP freeing the CPU for this task.
References[edit | edit source]
- NXP gstreamer' plugin github gstreamer-imx
- NPX AN13829 How to Use PXP in GStreamer and Wayland
