Media (Intel® oneAPI Video Processing Library, Intel Media SDK)
Access community support with transcoding, decoding, and encoding in applications using media tools from Intel. This includes Intel® oneAPI Video Processing Library and Intel® Media SDK.
Announcements
Welcome to the Intel Community. If you get an answer you like, please mark it as an Accepted Solution to help others. Thank you!
2905 Discussions

pipeline with 'mfxh264dec' gives larger end-to-end latency than 'avh264_dec'

Jiang__Dou
Beginner
260 Views

Problem:

With the same H264 RTP sender stream,  receiver pipeline with 'mfxh264dec' gives larger latency (30~60ms) than pipeline with ‘avdec_h264.

In my application, end-to-end latency is critical.

Since mfxh264dec with HW acceleration has less CPU usage than avdec_h264,

it would be great if mfxh264dec could give the (almost) same latency as avdec_264.


Plarform: 
Win10,  VS2017,  gstreamer-1.16,  Media SDK 2017
gstreamer MSDK plugin from: 
https://github.com/intel/gstreamer-media-SDK.git
(Built with OpenGL support using VS2017)

Note 'mfxh264dec' is built WITH OpenGl support:

C:\Users\UI>gst-inspect-1.0 --gst-plugin-load=C:\gst_MSDK_intel\gst-mfx-build_msvc\gst\mfx\gstmfx.dll mfxh264dec
Factory Details:
  Rank                     primary + 3 (259)
  Long-name                MFX H264 decoder
  Klass                    Codec/Decoder/Video
  Description              An MFX-based H264 video decoder
  Author                   Ishmael Sameen<ishmael.visayana.sameen@intel.com>

Plugin Details:
  Name                     mfx
  Description              MFX encoder/decoder/video post-processing plugins
  Filename                 C:\gst_MSDK_intel\gst-mfx-build_msvc\gst\mfx\gstmfx.dll
  Version                  2.0.2
  License                  LGPL
  Source module            gst_mfx
  Binary package           gst_mfx
  Origin URL               http://www.intel.com

GObject
 +----GInitiallyUnowned
       +----GstObject
             +----GstElement
                   +----GstVideoDecoder
                         +----GstMfxDec_h264

Pad Templates:
  SINK template: 'sink'
    Availability: Always
    Capabilities:
      video/x-h264
              alignment: au
                profile: { (string)constrained-baseline, (string)baseline, (string)main, (string)high }
          stream-format: byte-stream
  SRC template: 'src'
    Availability: Always
    Capabilities:
      video/x-raw(memory:MFXSurface)
                 format: { (string)NV12, (string)BGRA, (string)P010_10LE, (string)YUY2, (string)ENCODED }
                  width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]
              framerate: [ 0/1, 2147483647/1 ]
      video/x-raw(memory:GLMemory)
                 format: { (string)RGBA }
                  width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]
              framerate: [ 0/1, 2147483647/1 ]
      video/x-raw
                 format: { (string)NV12, (string)BGRA, (string)P010_10LE, (string)YUY2 }
                  width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]
              framerate: [ 0/1, 2147483647/1 ]


Here are sender and receiver pipelines:

1. H264 RTP ender pipeline on Nvidia TX2:

gst-launch-1.0 v4l2src device=/dev/video0 ! 'video/x-raw, format=UYVY, framerate=30/1' ! \
        nvvidconv ! 'video/x-raw(memory:NVMM),format=I420, framerate=30/1' ! \
        omxh264enc bitrate=$bitrate MeasureEncoderLatency=true ! 'video/x-h264, stream-format=(string)byte-stream' ! \
        rtph264pay ! udpsink host=$ip port=$port sync=false async=false

Using wireshark, I confirmed that the sps.pic_order_cnt_type is 2.


2. Receiver pipeline with avdec_h264:

gst-launch-1.0   udpsrc port=5000 caps="application/x-rtp, encoding-name=H264,payload=96" !  \
                       rtph264depay ! h264parse ! avh264_dec ! glimagesink sync=false async=false

3. Receiver pipeline with mfxh264dec:

gst-launch-1.0  --gst-plugin-load=C:\gst_MSDK_intel\gst-mfx-build_msvc\gst\mfx\gstmfx.dll udpsrc port=5000 caps="application/x-rtp, encoding-name=H264,payload=96" !  \
                       rtph264depay ! h264parse ! mfxh264dec live-mode=true ! glimagesink sync=false async=false

I found the latency topic was discussed:
https://software.intel.com/en-us/forums/intel-media-sdk/topic/704136
And in this discussion,  Dmitry E. (Intel) Thu, 06/22/2017 - 14:37 gave possible solutions.


Now here if 'live-mode=true',

In gst-libs/mfx/gstmfxdecoder.c, line 333~line 339:

 if (live_mode) {
    decoder->params.AsyncDepth = 1;
    decoder->bs.DataFlag = MFX_BITSTREAM_COMPLETE_FRAME;
    /* Hack for H264 low-latency streaming */
    if (decoder->params.mfx.CodecId == MFX_CODEC_AVC)
      decoder->params.mfx.DecodedOrder = 1;
  }

And remember sender's sps.pic_order_cnt_type ==  2.

So ALL possible solutions have been applied,
but the latency is still there.

Any suggestion?

0 Kudos
2 Replies
Dmitry_E_Intel
Employee
260 Views

I believe the right sequence can be the following:

1) Please submit a bug against https://github.com/intel/gstreamer-media-SDK/

2) If 'mfxh264dec' maintainers confirm it's a MSDK issue, then the issue can be filled against https://github.com/Intel-Media-SDK/MediaSDK

Mark_L_Intel1
Moderator
260 Views

Hi Dou,

I noticed you are running decoding from RTSP stream behind GStreamer plugin, could you double check it with sample_decode.exe?

To do this, you have to input the raw stream from a file and then set up DepthAsync and DecodeOrder manually by following the example in https://software.intel.com/en-us/forums/intel-media-sdk/topic/704136

Mark

Reply