XUELK-AN-007: USB Camera input on i.MX6UL-based platforms

From DAVE Developer's Wiki
Jump to: navigation, search
Info Box
AXEL ULite-top.png Applies to AXEL ULite
Warning-icon.png This application note was validated against specific versions of the kit only. It may not work with other versions. Supported versions are listed in the History section. Warning-icon.png

History[edit | edit source]

Version Date XUELK version Notes

1.0.0

July 2017 1.1.3 First release

Introduction[edit | edit source]

As known, i.MX6UL SoC has not an IPU cell allowing strong performances on video acquisition. However, using an USB camera it is possible to capture a raw data video playing it on LCD controller.

This application note describes some tests that were performed on an AXELULite-based platform to evaluate the capabilities and the performances of the i.MXUL when it captures and displays a camera source on LCD panel. The test bed was based on Yocto distribution using smart package manager for adding required software component.

USB camera driver[edit | edit source]

Common USB cameras can be used with a standard USB Video Class driver. Linux kernel must be compiled with this driver built-in or the kernel module can be dynamically loaded (in this case it has to be available on /lib/modules root file system directory).

Once plugged on USB host connector, the kernel driver recognizes the device and create the device node /dev/video1

[    6.989549] uvcvideo: Found UVC 1.00 device USB2.0 Camera (1871:0141)
[    7.041576] input: USB2.0 Camera as /devices/soc0/soc.0/2100000.aips-bus/2184200.usb/ci_hdrc.1/usb2/2-1/2-1:1.0/input/input5
[    7.138819] usbcore: registered new interface driver uvcvideo
[    7.191864] USB Video Class driver (1.1.1)
[    7.439394] usb_audio: Warning! Unlikely big volume range (=6144), cval->res is probably wrong.
[    7.489486] usb_audio: [3] FU [Mic Capture Volume] ch = 1, val = 5120/11264/1
[    7.546073] usbcore: registered new interface driver snd-usb-audio

gstreamer[edit | edit source]

Starting from original Yocto based root file system provided here, more packages need to be added for using the well known multimedia framework gstreamer

More info about gstreamer can be found here and on Github gstreamer server


We assume that network interface has been already configured for Internet access. In any case, simple configuration can be done directly on serial console with the following commands:

root@sbc-lynx:~# ifconfig eth0 192.168.0.94
root@sbc-lynx:~# route add default gw 192.168.0.254
root@sbc-lynx:~# echo "nameserver 8.8.8.8" > /etc/resolv.conf

where 192.168.0.254 is the gateway address and 8.8.8.8 is the DNS server (in this case Google's one).

smart packages[edit | edit source]

Firts of all, SMART channels must be added for accessing DAVE's server for Yocto pre-built packages, using the following commands:

root@sbc-lynx:~# smart channel --add cortexa7hf_vfp_neon type=rpm-md baseurl=http://yocto.dave.eu/xuelk-latest/cortexa7hf_vfp_neon
root@sbc-lynx:~# smart channel --add cortexa7hf_vfp_neon_mx6ul type=rpm-md baseurl=http://yocto.dave.eu/xuelk-latest/cortexa7hf_vfp_neon_mx6ul
root@sbc-lynx:~# smart channel --add sbc_lynx type=rpm-md baseurl=http://yocto.dave.eu/xuelk-latest/sbc_lynx

and finally updating the smart cache checking for the new available packages:

root@sbc-lynx:~# smart update
Updating cache...               ######################################## [100%]

Fetching information for 'cortexa7hf_vfp_neon_mx6ul'...
-> http://yocto.dave.eu/xuelk-latest/cortexa7hf_vfp_neon_mx6ul/.../repomd.xml
repomd.xml                      ######################################## [ 16%]
-> http://yocto.dave.eu/xuelk-latest/.../filelists.xml.gz
filelists.xml.gz                ######################################## [ 25%]
-> http://yocto.dave.eu/xuelk-latest/.../primary.xml.gz
primary.xml.gz                  ######################################## [ 33%]

Fetching information for 'cortexa7hf_vfp_neon'...
-> http://yocto.dave.eu/xuelk-latest/cortexa7hf_vfp_neon/repodata/repomd.xml
repomd.xml                      ######################################## [ 50%]
-> http://yocto.dave.eu/xuelk-latest/cortexa7hf_vfp_neon/.../filelists.xml.gz
filelists.xml.gz                ######################################## [ 58%]
-> http://yocto.dave.eu/xuelk-latest/cortexa7hf_vfp_neon/.../primary.xml.gz
primary.xml.gz                  ######################################## [ 66%]

Fetching information for 'sbc_lynx'...
-> http://yocto.dave.eu/xuelk-latest/sbc_lynx/repodata/repomd.xml
repomd.xml                      ######################################## [ 83%]
-> http://yocto.dave.eu/xuelk-latest/sbc_lynx/repodata/filelists.xml.gz
-> http://yocto.dave.eu/xuelk-latest/sbc_lynx/repodata/primary.xml.gz
primary.xml.gz                  ######################################## [ 91%]
filelists.xml.gz                ######################################## [100%]

Updating cache...               ######################################## [100%]

Channels have 11874 new packages.
Saving cache...

root@sbc-lynx:~#

Adding these three channels there will be 11874 new available packages.

Adding gstreamer packages[edit | edit source]

It is possibile to install the gstreamer 1.0 tool using the following command:

root@sbc-lynx:~# smart install gstreamer1.0-1.4.5-r0
Loading cache...
Updating cache...               ######################################## [100%]

Computing transaction...

Installing packages (1):
  gstreamer1.0-1.4.5-r0@cortexa7hf_vfp_neon

763.6kB of package files are needed. 1.8MB will be used.

Confirm changes? (Y/n): y

Fetching packages...
-> http://yocto.dave.eu/.../gstreamer1.0-1.4.5-r0.cortexa7hf_vfp_neon.rpm
gstreamer1.0-1.4.5-r0.cortexa.. ######################################## [100%]


Committing transaction...
Preparing...                    ######################################## [  0%]
   1:Installing gstreamer1.0    ######################################## [100%]


root@sbc-lynx:~#

Then, it is enough to install the other gstreamer plugins, for example:

gstreamer1.0-plugins-good-1.4.5-r0
gstreamer1.0-plugins-base-1.4.5-r0
gstreamer1.0-plugins-bad-1.4.5-r0

These packages will install about:

plugins-good: 5.5MB of package files are needed. 11.7MB will be used.
plugins-base: 1.8MB of package files are needed. 3.7MB will be used.
plugins-bad: 2.1MB of package files are needed. 3.9MB will be used.

Adding fsl gstreamer plugin[edit | edit source]

For video sink on framebuffer, the NXP gstreamer plugin is required. This can be installed from fsl package:

root@sbc-lynx:~# smart install gst1.0-fsl-plugin-4.0.8-r0
Loading cache...
Updating cache...               ######################################## [100%]

Computing transaction...

Installing packages (7):
  gst1.0-fsl-plugin-4.0.8-r0@sbc_lynx
  lib-oggvorbis-dec-arm11-elinux2-4.0.8-r0@cortexa7hf_vfp_neon_mx6ul
  libfslcodec-aac-4.0.8-r0@cortexa7hf_vfp_neon_mx6ul
  libfslcodec-mp3-4.0.8-r0@cortexa7hf_vfp_neon_mx6ul
  libfslparser-4.0.8-r0@cortexa7hf_vfp_neon_mx6ul
  libgstfsl-1.0-0-4.0.8-r0@sbc_lynx
  libpxp0-1:5.3-r0@sbc_lynx

1.6MB of package files are needed. 5.9MB will be used.

Confirm changes? (Y/n): y

Fetching packages...
-> http://yocto.dave.eu/.../libfslcodec-mp3-4.0.8-r0.cortexa7hf_vfp_neon_mx6ul.rpm
-> http://yocto.dave.eu/.../lib-oggvorbis-dec-arm11-elinux2-4.0.8-r0.cortexa7hf_vfp_neon_mx6ul.rpm
lib-oggvorbis-dec-arm11-elinu.. ######################################## [ 14%]
-> http://yocto.dave.eu/xuelk-latest/.../libgstfsl-1.0-0-4.0.8-r0.sbc_lynx.rpm
libgstfsl-1.0-0-4.0.8-r0.sbc_.. ######################################## [ 28%]
-> http://yocto.dave.eu/.../libfslcodec-aac-4.0.8-r0.cortexa7hf_vfp_neon_mx6ul.rpm
libfslcodec-aac-4.0.8-r0.cort.. ######################################## [ 42%]
libfslcodec-mp3-4.0.8-r0.cort.. ######################################## [ 57%]
-> http://yocto.dave.eu/.../gst1.0-fsl-plugin-4.0.8-r0.sbc_lynx.rpm
-> http://yocto.dave.eu/xuelk-latest/sbc_lynx/libpxp0-5.3-r0.sbc_lynx.rpm
libpxp0-5.3-r0.sbc_lynx.rpm     ######################################## [ 71%]
-> http://yocto.dave.eu/.../libfslparser-4.0.8-r0.cortexa7hf_vfp_neon_mx6ul.rpm
libfslparser-4.0.8-r0.cortexa.. ######################################## [ 85%]
gst1.0-fsl-plugin-4.0.8-r0.sb.. ######################################## [100%]


Committing transaction...
Preparing...                    ######################################## [  0%]
   1:Installing libfslparser    ######################################## [ 14%]
   2:Installing libfslcodec-mp3 ######################################## [ 28%]
   3:Installing libfslcodec-aac ######################################## [ 42%]
   4:Installing libpxp0         ######################################## [ 57%]
   5:Installing lib-oggvorbis.. ######################################## [ 71%]
   6:Installing libgstfsl-1.0-0 ######################################## [ 85%]
   7:Installing gst1.0-fsl-pl.. ######################################## [100%]


root@sbc-lynx:/lib/modules#

gstreamer pipeline[edit | edit source]

The pipeline used is the easiest one for capturing a raw data stream from a standard USB camera

gst-launch-1.0 -v v4l2src device=/dev/video1 ! imxv4l2sink

Test bed[edit | edit source]

  • iMX6UL cpu is running at 528MHz
root@arm:~# cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq
528000
  • system DRAM is 256MB
  • lcd panel is 800x480

Results[edit | edit source]

Use case #1: Video capture 640x480@30fps[edit | edit source]

  • pipeline:
root@sbc-lynx:~# gst-launch-1.0 -v v4l2src device=/dev/video1 ! 'video/x-raw, format=YUY2, width=640, height=480, framerate=30/1' ! imxv4l2sink force-aspect-ratio=true
====== IMXV4L2SINK: 4.0.8 build on Aug 25 2016 18:01:57. ======
Setting pipeline to PAUSED ...
display(/dev/fb0) resolution is (800x480).
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
/GstPipeline:pipeline0/GstV4l2Src:v4l2src0.GstPad:src: caps = "video/x-raw\,\ format\=\(string\)YUY2\,\ width\=\(int\)640\,\ height\=\(int\)480\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ interlace-mode\=\(string\)progressive\,\ framerate\=\(fraction\)30/1"
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = "video/x-raw\,\ format\=\(string\)YUY2\,\ width\=\(int\)640\,\ height\=\(int\)480\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ interlace-mode\=\(string\)progressive\,\ framerate\=\(fraction\)30/1"
/GstPipeline:pipeline0/GstImxV4l2Sink:imxv4l2sink0.GstPad:sink: caps = "video/x-raw\,\ format\=\(string\)YUY2\,\ width\=\(int\)640\,\ height\=\(int\)480\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ interlace-mode\=\(string\)progressive\,\ framerate\=\(fraction\)30/1"
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = "video/x-raw\,\ format\=\(string\)YUY2\,\ width\=\(int\)640\,\ height\=\(int\)480\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ interlace-mode\=\(string\)progressive\,\ framerate\=\(fraction\)30/1"
v4l2sink need allocate 3 buffers.
  • top process shows that cpu is running at 20%
Mem: 81752K used, 165412K free, 356K shrd, 9084K buff, 18600K cached
CPU:   2% usr  13% sys   0% nic  64% idle   0% io   0% irq  19% sirq
Load average: 0.13 0.09 0.05 1/64 452
  PID  PPID USER     STAT   VSZ %VSZ %CPU COMMAND
  432   386 root     S    35364  14%  22% gst-launch-1.0 -v v4l2src device=/dev/
  447   386 root     R     2828   1%   2% top -d 1
   71     2 root     SW       0   0%   1% [pxp_dispatch]
    3     2 root     SW       0   0%   1% [ksoftirqd/0]
  382     1 root     S     5028   2%   0% /usr/sbin/tcf-agent -d -L- -l0
...

Use case #2: Video capture 640x480@30fps scaled to 320x240[edit | edit source]

This use case is similar to the previous one, but the video data has been scaled to 1/4 using the imxv4l2sink plugin overlay capabilities

root@sbc-lynx:~# gst-launch-1.0 -v v4l2src device=/dev/video1 ! 'video/x-raw, format=YUY2, width=320, height=240, framerate=30/1' ! imxv4l2sink overlay-top=120 overlay-left=240 overlay-width=320 overlay-height=240 force-aspect-ratio=true
====== IMXV4L2SINK: 4.0.8 build on Aug 25 2016 18:01:57. ======
Setting pipeline to PAUSED ...
display(/dev/fb0) resolution is (800x480).
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
/GstPipeline:pipeline0/GstV4l2Src:v4l2src0.GstPad:src: caps = "video/x-raw\,\ format\=\(string\)YUY2\,\ width\=\(int\)320\,\ height\=\(int\)240\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ interlace-mode\=\(string\)progressive\,\ framerate\=\(fraction\)30/1"
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = "video/x-raw\,\ format\=\(string\)YUY2\,\ width\=\(int\)320\,\ height\=\(int\)240\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ interlace-mode\=\(string\)progressive\,\ framerate\=\(fraction\)30/1"
/GstPipeline:pipeline0/GstImxV4l2Sink:imxv4l2sink0.GstPad:sink: caps = "video/x-raw\,\ format\=\(string\)YUY2\,\ width\=\(int\)320\,\ height\=\(int\)240\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ interlace-mode\=\(string\)progressive\,\ framerate\=\(fraction\)30/1"
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = "video/x-raw\,\ format\=\(string\)YUY2\,\ width\=\(int\)320\,\ height\=\(int\)240\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ interlace-mode\=\(string\)progressive\,\ framerate\=\(fraction\)30/1"
v4l2sink need allocate 3 buffers.
  • top process shows that cpu is running at 13%
Mem: 176956K used, 70208K free, 356K shrd, 3896K buff, 116504K cached
CPU:   0% usr   9% sys   0% nic  71% idle   0% io   0% irq  19% sirq
Load average: 0.27 0.35 0.36 1/64 683
  PID  PPID USER     STAT   VSZ %VSZ %CPU COMMAND
  676   378 root     S    35512  14%  13% gst-launch-1.0 -v v4l2src device=/dev/

Use case #3: Video capture 640x480@30fps with rotation[edit | edit source]

In this use case the PXP iMX6UL hardware cell is performing an image rotation on the screen using the imxvideoconvert_pxp NXP gstreamer plugin

root@sbc-lynx:~# gst-launch-1.0 -v v4l2src device=/dev/video1 ! capsfilter caps="video/x-raw, width=640, height=480, framerate=30/1" ! imxvideoconvert_pxp rotation=1 ! imxv4l2sink force-aspect-ratio=true
====== IMXV4L2SINK: 4.0.8 build on Aug 25 2016 18:01:57. ======
Setting pipeline to PAUSED ...
display(/dev/fb0) resolution is (800x480).
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
/GstPipeline:pipeline0/GstV4l2Src:v4l2src0.GstPad:src: caps = "video/x-raw\,\ format\=\(string\)YUY2\,\ width\=\(int\)640\,\ height\=\(int\)480\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ interlace-mode\=\(string\)progressive\,\ framerate\=\(fraction\)30/1"
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = "video/x-raw\,\ format\=\(string\)YUY2\,\ width\=\(int\)640\,\ height\=\(int\)480\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ interlace-mode\=\(string\)progressive\,\ framerate\=\(fraction\)30/1"
/GstPipeline:pipeline0/imxvideoconvert_pxp:imxvideoconvert_pxp0.GstPad:src: caps = "video/x-raw\,\ format\=\(string\)BGRx\,\ width\=\(int\)640\,\ height\=\(int\)480\,\ framerate\=\(fraction\)30/1\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ interlace-mode\=\(string\)progressive"
/GstPipeline:pipeline0/GstImxV4l2Sink:imxv4l2sink0.GstPad:sink: caps = "video/x-raw\,\ format\=\(string\)BGRx\,\ width\=\(int\)640\,\ height\=\(int\)480\,\ framerate\=\(fraction\)30/1\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ interlace-mode\=\(string\)progressive"
/GstPipeline:pipeline0/imxvideoconvert_pxp:imxvideoconvert_pxp0.GstPad:sink: caps = "video/x-raw\,\ format\=\(string\)YUY2\,\ width\=\(int\)640\,\ height\=\(int\)480\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ interlace-mode\=\(string\)progressive\,\ framerate\=\(fraction\)30/1"
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = "video/x-raw\,\ format\=\(string\)YUY2\,\ width\=\(int\)640\,\ height\=\(int\)480\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ interlace-mode\=\(string\)progressive\,\ framerate\=\(fraction\)30/1"
v4l2sink need allocate 3 buffers.

Demo on youtube[edit | edit source]

Video capturing and streaming on NXP i MX6 UL