OpenVINO using OpenCV with Gstreamer - need pipeline help

[DaveK]

I am looking for some suggestions and help on getting my Gstreamer pipeline to work on my Intel NUC8i5 to use hardware-accelerated video decode and then encode for C++ motion detection program. 

I have installed OpenVINO 2021.1;  its complete build information is shown in the OpenVINO_OpenCV .rtf attached file.

General configuration for OpenCV 4.5.0-openvino =====================================
Version control: 464efeb7b07211706d0f2615e16c8d144447d362

Timestamp: 2020-09-29T02:20:48Z
Host: Linux 4.15.0-29-generic x86_64
CMake: 3.14.5
CMake generator: Ninja
CMake build tool: /opt/miniconda/envs/build/bin/ninja
Configuration: Release

Here is the snippet of C++ code which attempts to capture the video stream from IP camera using RTSP and Gstreamer:

std::string gst_pipe_in = "rtspsrc location=rtsp://admin:pswd@192.168.1.33 ! rtph264depay ! h264parse ! vaapih264dec ! autovideoconvert ! appsink";
cv::VideoCapture capture(gst_pipe_in,cv::CAP_GSTREAMER);

Once compiled and run, I can't get it to work. 

If I don't use Gstreamer, it works fine, but I want to use hardware decode and encode.

cv::VideoCapture capture("rtsp://admin:pswd@192.168.1.33");

Here is the output when I run the program:

dk@NUC8i5:~$ GST_DEBUG=*:3 ./t3

Welcome ~ NUC 8i5 ~ C++ Cul-de-Sac Motion Detection

=== Program STARTED at 2020-12-09 12:45:13 ===

OpenCV version: 4.5.0-openvino

0:00:00.016150233 1732 0x563b1f70dc90 WARN autoconvert gstautoconvert.c:1048:gst_auto_convert_sink_query:<autoconvertchild> Got query context while no element was selected, letting through
0:00:00.307872696 1732 0x7f2b08044230 FIXME default gstutils.c:3981:gst_pad_create_stream_id_internal:<fakesrc0:src> Creating random stream-id, consider implementing a deterministic way of creating a stream-id
0:00:01.183670669 1732 0x7f2aec003b70 FIXME basesink gstbasesink.c:3248:gst_base_sink_default_event:<appsink0> stream-start event without group-id. Consider implementing group-id handling in the upstream elements
0:00:01.183930200 1732 0x7f2aec003b70 WARN basetransform gstbasetransform.c:1364:gst_base_transform_setcaps:<vaapipostproc0> transform could not transform video/x-raw(memory:VASurface), format=(string)NV12, width=(int)2688, height=(int)1520, interlace-mode=(string)progressive, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)12/1 in anything we support

ANY help?  I can provide more info on system, program, installation, etc.

THANK YOU so much!

Dave

[Maksim_S_Intel]

Try to use videoconvert (base) instead of autovideoconvert (bad). Try to add caps after decoder: video/x-raw,format=NV12. Try to use vaapidecodebin, maybe with disable-vpp flag enabled.

Set environment variable GST_DEBUG_DUMP_DOT_DIR=<dir> (dir should exist) and check DOT graphs pipeline-init and pipeline-error for hints.

What is your OS and libva version? Does vainfo command work? There were problems with GStreamer/VA-API decoding on older Ubuntu 16, while Ubuntu 18 worked much better (system packages).

Hint: try to add sync=false option to your appsink, then there will be no synchronization with stream FPS and processing can be faster.

[DaveK]

Maksim_S_Intel,

THANK YOU for your tips.   I have got it working, yet with WARNINGS.

I am running Ubuntu 18.04.5 LTS.

My code as of now:

std::string gst_pipe_in = "rtspsrc location=rtsp://admin:pswd@192.168.1.33 ! rtph264depay ! h264parse ! vaapih264dec ! video/x-raw,format=NV12,width=2688,height=1520,framerate=12/1 ! videoconvert ! appsink sync=false";

cv::VideoCapture capture(gst_pipe_in,cv::CAP_GSTREAMER);

Here are some ERRORS and WARNINGS... do I need to worry about these, as the program seems to be working.

dk@NUC8i5:~$ GST_DEBUG=*:3 ./g1

Welcome ~ NUC 8i5 ~  C++  Cul-de-Sac Motion Detection

=== Program STARTED at 2020-12-14 18:38:49 ===

OpenCV version: 4.5.0-openvino

0:00:00.468402425  2502 0x7fb0dc040e30 FIXME              default gstutils.c:3981:gst_pad_create_stream_id_internal:<fakesrc0:src> Creating random stream-id, consider implementing a deterministic way of creating a stream-id

0:00:00.620426052  2502 0x7fb0bc003770 FIXME              basesink gstbasesink.c:3248:gst_base_sink_default_event:<appsink0> stream-start event without group-id. Consider implementing group-id handling in the upstream elements

[ WARN:0] global ../opencv/modules/videoio/src/cap_gstreamer.cpp (898) open OpenCV | GStreamer warning: unable to query duration of stream

[ WARN:0] global ../opencv/modules/videoio/src/cap_gstreamer.cpp (935) open OpenCV | GStreamer warning: Cannot query video position: status=1, value=0, duration=-1

Initializing IP Camera Video Stream

Entering Video Capture Main Processing Loop

OpenCV: FFMPEG: tag 0x00000708/'????' is not found (format 'mp4 / MP4 (MPEG-4 Part 14)')'

0:00:06.202577576  2502 0x7fb0dc00a1e0 FIXME              default gstutils.c:3981:gst_pad_create_stream_id_internal:<appsrc0:src> Creating random stream-id, consider implementing a deterministic way of creating a stream-id

0:00:06.246634222  2502 0x560d20725de0 FIXME              basesink gstbasesink.c:3248:gst_base_sink_default_event:<filesink0> stream-start event without group-id. Consider implementing group-id handling in the upstream elements

0:00:06.246682411  2502 0x560d20725de0 WARN                   qtmux gstqtmux.c:2982:gst_qt_mux_start_file:<mp4mux0> Robust muxing requires reserved-moov-update-period to be set

My output code:

std::string motion_writer_pipe = "appsrc ! videoconvert ! vaapih264enc bitrate=8000 ! video/x-h264,profile=main ! h264parse ! mp4mux ! filesink location = " + savedMotionVideoFullPath;

THANKS AGAIN for your help!

Dave

[Iffa_Intel]

Hi,

Glad to know that it works for you.

It is fine to receive Warning indication as long as no error occurs and the program works.

Is there anything else that you require help with? otherwise, I will proceed to close this thread

Sincerely,

Iffa

[Iffa_Intel]

Greetings,

Intel will no longer monitor this thread since we have provided a solution. If you need any additional information from Intel, please submit a new question

Sincerely,

Iffa