vaapipostproc in opencv

[eskay]

Hi

I am trying to use vaapih264dec and vaapipostprocess for getting frames in to opencv with following code.

video_capture = cv2.VideoCapture("rtspsrc location={} ! rtph264depay ! h264parse ! vaapih264dec ! vaapipostproc ! appsink sync=false".format("rtsp://ip:port/test"), cv2.CAP_GSTREAMER)

But it is stuck, doesnt do anything.

But if i use videoconvert instead of vaapipostproc, it works but cpu utilization is bit high.
video_capture = cv2.VideoCapture("rtspsrc location={} ! rtph264depay ! h264parse ! vaapih264dec ! videoconvert ! appsink sync=false".format("rtsp://ip:port/test"), cv2.CAP_GSTREAMER)

Reason for using vaapipostproc is to save cpu compute for color space conversion of vaapih264dec result.

However, when the vaapipostproce used in terminal, i observed that it is able to run smootlhy.

gst-launch-1.0 rtspsrc location=rtsp://ip:port/test ! rtph264depay ! h264parse ! vaapih264dec ! vaapipostproc ! fakesink --> (works with very less cpu consumption) cpu is 4-6%

To confirm, wrote images to filesystem as followes for both (vaapipostproc and videoconvert). Both worked and written images to disk.

gst-launch-1.0 rtspsrc location=rtsp://ip:port/test ! rtph264depay ! h264parse ! vaapih264dec ! vaapipostproc ! jpegenc ! multifilesink location="t%06d.jpeg"

gst-launch-1.0 rtspsrc location=rtsp://ip:port/test ! rtph264depay ! h264parse ! vaapih264dec ! videoconvert ! jpegenc ! multifilesink location="t%06d.jpeg"

Tried videoconvert after vaapipostprocm but cpu consumption is comparable to videoconvert alone.
video_capture = cv2.VideoCapture("rtspsrc location={} ! rtph264depay ! h264parse ! vaapih264dec ! vaapipostproc ! videoconvert ! appsink sync=false".format("rtsp://ip:port/test"), cv2.CAP_GSTREAMER)

How to use vaapipostproc without videoconvert in opencv to consume frames.
GST versio 1.16.2
docker image: openvino/ubuntu20_data_dev:2021.3_src
HW: Intel i7 10th gen
driver: iHD graphics inside openvino docker image

[David_G_Intel]

Hello eskay

Thank you for posting on the Intel️® communities. We will move your question to the correct sub forum, the team in charge will get back to you soon.

Regards, 

David G 

Intel Customer Support Technician 

[Maksim_S_Intel · Accepted solution]

Perhaps vaapipostproc can not convert ANY supported input format to ANY supported output format (see link), only specific combinations. OpenCV requests BGR/GRAY/Bayer/jpeg output (it is like video/x-raw,format=BGR filter before/on appsink element, see link).

In my experiment vaapipostproc could not convert NV12 to BGR (i965 driver from Ubuntu 18):

$ gst-launch-1.0 videotestsrc num-buffers=5 ! video/x-raw,format=NV12 ! vaapipostproc ! video/x-
raw,format=BGR ! fakesink
Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
Got context from element 'vaapipostproc0': gst.gl.GLDisplay=context, gst.gl.GLDisplay=(GstGLDisplay)"\(GstGLDisplayGBM\)\ gldisplaygbm0";
Got context from element 'vaapipostproc0': gst.vaapi.Display=context, gst.vaapi.Display=(GstVaapiDisplay)"\(GstVaapiDisplayDRM\)\ vaapidisplaydrm1";
ERROR: from element /GstPipeline:pipeline0/GstVideoTestSrc:videotestsrc0: Internal data stream error.
Additional debug info:
gstbasesrc.c(3055): gst_base_src_loop (): /GstPipeline:pipeline0/GstVideoTestSrc:videotestsrc0:
streaming stopped, reason not-negotiated (-4)
ERROR: pipeline doesn't want to preroll.
Setting pipeline to NULL ...
Freeing pipeline ...

And it got no problem converting NV12 to YV12, BGRA, RGBA:

$ gst-launch-1.0 videotestsrc num-buffers=5 ! video/x-raw,format=NV12 ! vaapipostproc ! video/x-
raw,format=YV12 ! fakesink
Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
Got context from element 'vaapipostproc0': gst.gl.GLDisplay=context, gst.gl.GLDisplay=(GstGLDisplay)"\(GstGLDisplayGBM\)\ gldisplaygbm0";
Got context from element 'vaapipostproc0': gst.vaapi.Display=context, gst.vaapi.Display=(GstVaapiDisplay)"\(GstVaapiDisplayDRM\)\ vaapidisplaydrm1";
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
Got EOS from element "pipeline0".
Execution ended after 0:00:00.001256660
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...

 In your successful attempts you used jpegenc which supports various input formats (see link) including NV12 incoming from decoder and fakesink which support any input format, so no actual conversion have been performed.

I can recommend the following:

* consult on Media Forum (the one from which this topic has been moved here) on media drivers and supported color conversion combinations.

* use videoconvert and tune its CPU usage with the n-threads property.

* try to use vaapipostproc to convert to BGRA and then videoconvert to convert to BGR, maybe it will not use so much CPU resources in this case.

Another option is to add BGRA input support to OpenCV GStreamer videoio backend and perform conversion using OpenCV functions internally or on user's side.

[eskay]

Thanks @Maksim_S_Intel  for the detailed explanantion about the pluggins.

But to my surprise, vaapih264dec has RGB output in video/x-raw tag

video/x-raw:

(https://gstreamer.freedesktop.org/documentation/vaapi/vaapih264dec.html?gi-language=c) but when used pipeline doesnt get created.

[Maksim_S_Intel]

Perhaps RGB output is listed there formally (maybe libva allows it) but is not supported by the media driver (iHD): https://github.com/intel/media-driver/blob/master/docs/media_features.md#supported-decoding-output-format-and-max-resolution

I believe vaapi plugin detects capabilities dynamically, to see them run "gst-inspect-1.0 vaapih264dec".

[Maksim_S_Intel]

I've experimented a bit more with various image formats and looks like the fastest and most efficient way is to get NV12 from decoder and pass it to cvtColor to make BGR.

Pipeline is as follows: "filesrc location={} ! qtdemux ! vaapih264dec ! video/x-raw, format=NV12 ! appsink sync=false" + cvtColor(frame, COLOR_YUV2BGR_NV12), Linux, pthreads backend

• 1 thread:
  Read: 17532 frames (28.190231561660767 sec)
  22.71user 3.23system 0:28.50elapsed 91%CPU (0avgtext+0avgdata 122532maxresident)k
• 2 threads:
  Read: 17532 frames (19.85002374649048 sec)
  29.38user 5.87system 0:20.15elapsed 174%CPU (0avgtext+0avgdata 123156maxresident)k
• 3 threads:
  Read: 17532 frames (19.221385955810547 sec)
  42.77user 8.43system 0:19.53elapsed 262%CPU (0avgtext+0avgdata 125856maxresident)k
• 4 threads:
  Read: 17532 frames (19.190452575683594 sec)
  57.55user 10.83system 0:19.50elapsed 350%CPU (0avgtext+0avgdata 122164maxresident)k
• 8 threads:
  Read: 17532 frames (21.68948793411255 sec)
  118.37user 26.49system 0:22.01elapsed 657%CPU (0avgtext+0avgdata 123932maxresident)k

Using vaapipostproc to convert to BGRA[1] and then cvtColor to convert to BGR is slower:  "filesrc location={} ! qtdemux ! vaapih264dec ! vaapipostproc ! video/x-raw, format=BGRA ! appsink sync=false" + cvtColor(frame, COLOR_BGRA2BGR), Linux, pthreads backend

• 1 thread:
  Read: 17532 frames (55.87683176994324 sec)
  41.54user 3.85system 0:56.22elapsed 80%CPU (0avgtext+0avgdata 154072maxresident)k
• 2 threads:
  Read: 17532 frames (53.54783797264099 sec)
  55.63user 7.28system 0:53.89elapsed 116%CPU (0avgtext+0avgdata 153292maxresident)k

• 8 threads:
  Read: 17532 frames (60.107177734375 sec)
  245.43user 58.00system 1:00.44elapsed 502%CPU (0avgtext+0avgdata 163268maxresident)k

Of course results can vary depending on HW/SW platform, video format and usage scenario, so it is better to check your specific use case.

Notes:

[1] - with patched version of OpenCV

[AthiraM_Intel]

Hi eskay,

Could you please give us an update? Has the solution provided by Maksim_S_Intel helped?

Thanks

[AthiraM_Intel]

Hi,

We assume that your issue is resolved. If you need any additional information, please submit a new question as this thread will no longer be monitored

Thanks.