Video mixing acceleration using gstreamer with VAAPI.

JBold3 · ‎03-17-2020

Hello,

We are using a UpSquared board with an Intel Atom E3950. We are also using the system below.

Linux upsquared 5.3.0-42-generic #34~18.04.1-Ubuntu SMP Fri Feb 28 13:42:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

We would like to capture uncompressed video sources and mix them in order to output the result through HDMI and DisplayPort. We intend to use gstreamer in order to capture from UVC devices and would like to use VAAPI in order to do the mixing. Here is the custom pipeline I am using:

gst-launch-1.0 compositor name=comp \
sink_0::xpos=0 sink_0::ypos=0 sink_0::width=1920 sink_0::height=1080 sink_0::zorder=0 \
sink_1::xpos=1920 sink_1::ypos=0 sink_1::width=1920 sink_1::height=1080 sink_1::zorder=0 \
sink_2::xpos=0 sink_2::ypos=1080 sink_2::width=1920 sink_2::height=1080 sink_2::zorder=0 \
sink_3::xpos=1920 sink_3::ypos=1080 sink_3::width=1920 sink_3::height=1080 sink_3::zorder=0 \
! queue ! fpsdisplaysink video-sink="vaapisink fullscreen=true" sync=false text-overlay=false \
v4l2src device=/dev/video0 io-mode=dmabuf ! 'video/x-raw,width=1920,height=1080,framerate=30/1' ! queue ! comp. \
v4l2src device=/dev/video8 io-mode=dmabuf ! 'video/x-raw,width=1920,height=1080,framerate=30/1' ! queue ! comp. \
v4l2src device=/dev/video2 io-mode=dmabuf ! image/jpeg,width=1920,height=1080,framerate=30/1 ! vaapijpegdec ! queue ! comp. \
v4l2src device=/dev/video6 io-mode=dmabuf ! image/jpeg,width=1920,height=1080,framerate=30/1 ! vaapijpegdec ! queue ! comp. -v

The result of this pipeline leads video stutters and I am not able to render video at 30fps. It renders a non stable video.

If I do the same setup but each video is rendered on 4 separate windows, the video is really more stable. It seems to be related to the "compositor" structure that is not hardware accelerated and is managed by CPU.

I also tried videomixer and glvideomixer and these one seems to have problems too.

Is there a way to mix 4 video sources and use VAAPI or any hardware accelerated structure in order to output it to a display?

CarlosAM_INTEL · ‎03-17-2020

Hello, @JBold3:

Thank you for contacting Intel Embedded Community.

You consultation of the device that you have mentioned should be addressed as a reference to the following channel:

https://forum.up-community.org/categories/up-square-linux

Best regards,

@Mæcenas_INTEL.