Media (Intel® Video Processing Library, Intel Media SDK)
Access community support with transcoding, decoding, and encoding in applications using media tools like Intel® oneAPI Video Processing Library and Intel® Media SDK
Announcements
The Intel Media SDK project is no longer active. For continued support and access to new features, Intel Media SDK users are encouraged to read the transition guide on upgrading from Intel® Media SDK to Intel® Video Processing Library (VPL), and to move to VPL as soon as possible.
For more information, see the VPL website.
3056 Discussions

pipeline with 'mfxh264dec' gives larger end-to-end latency than 'avh264_dec'

Jiang__Dou
Beginner
680 Views

Problem:

With the same H264 RTP sender stream,  receiver pipeline with 'mfxh264dec' gives larger latency (30~60ms) than pipeline with ‘avdec_h264.

In my application, end-to-end latency is critical.

Since mfxh264dec with HW acceleration has less CPU usage than avdec_h264,

it would be great if mfxh264dec could give the (almost) same latency as avdec_264.


Plarform: 
Win10,  VS2017,  gstreamer-1.16,  Media SDK 2017
gstreamer MSDK plugin from: 
https://github.com/intel/gstreamer-media-SDK.git
(Built with OpenGL support using VS2017)

Note 'mfxh264dec' is built WITH OpenGl support:

C:\Users\UI>gst-inspect-1.0 --gst-plugin-load=C:\gst_MSDK_intel\gst-mfx-build_msvc\gst\mfx\gstmfx.dll mfxh264dec
Factory Details:
  Rank                     primary + 3 (259)
  Long-name                MFX H264 decoder
  Klass                    Codec/Decoder/Video
  Description              An MFX-based H264 video decoder
  Author                   Ishmael Sameen<ishmael.visayana.sameen@intel.com>

Plugin Details:
  Name                     mfx
  Description              MFX encoder/decoder/video post-processing plugins
  Filename                 C:\gst_MSDK_intel\gst-mfx-build_msvc\gst\mfx\gstmfx.dll
  Version                  2.0.2
  License                  LGPL
  Source module            gst_mfx
  Binary package           gst_mfx
  Origin URL               http://www.intel.com

GObject
 +----GInitiallyUnowned
       +----GstObject
             +----GstElement
                   +----GstVideoDecoder
                         +----GstMfxDec_h264

Pad Templates:
  SINK template: 'sink'
    Availability: Always
    Capabilities:
      video/x-h264
              alignment: au
                profile: { (string)constrained-baseline, (string)baseline, (string)main, (string)high }
          stream-format: byte-stream
  SRC template: 'src'
    Availability: Always
    Capabilities:
      video/x-raw(memory:MFXSurface)
                 format: { (string)NV12, (string)BGRA, (string)P010_10LE, (string)YUY2, (string)ENCODED }
                  width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]
              framerate: [ 0/1, 2147483647/1 ]
      video/x-raw(memory:GLMemory)
                 format: { (string)RGBA }
                  width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]
              framerate: [ 0/1, 2147483647/1 ]
      video/x-raw
                 format: { (string)NV12, (string)BGRA, (string)P010_10LE, (string)YUY2 }
                  width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]
              framerate: [ 0/1, 2147483647/1 ]


Here are sender and receiver pipelines:

1. H264 RTP ender pipeline on Nvidia TX2:

gst-launch-1.0 v4l2src device=/dev/video0 ! 'video/x-raw, format=UYVY, framerate=30/1' ! \
        nvvidconv ! 'video/x-raw(memory:NVMM),format=I420, framerate=30/1' ! \
        omxh264enc bitrate=$bitrate MeasureEncoderLatency=true ! 'video/x-h264, stream-format=(string)byte-stream' ! \
        rtph264pay ! udpsink host=$ip port=$port sync=false async=false

Using wireshark, I confirmed that the sps.pic_order_cnt_type is 2.


2. Receiver pipeline with avdec_h264:

gst-launch-1.0   udpsrc port=5000 caps="application/x-rtp, encoding-name=H264,payload=96" !  \
                       rtph264depay ! h264parse ! avh264_dec ! glimagesink sync=false async=false

3. Receiver pipeline with mfxh264dec:

gst-launch-1.0  --gst-plugin-load=C:\gst_MSDK_intel\gst-mfx-build_msvc\gst\mfx\gstmfx.dll udpsrc port=5000 caps="application/x-rtp, encoding-name=H264,payload=96" !  \
                       rtph264depay ! h264parse ! mfxh264dec live-mode=true ! glimagesink sync=false async=false

I found the latency topic was discussed:
https://software.intel.com/en-us/forums/intel-media-sdk/topic/704136
And in this discussion,  Dmitry E. (Intel) Thu, 06/22/2017 - 14:37 gave possible solutions.


Now here if 'live-mode=true',

In gst-libs/mfx/gstmfxdecoder.c, line 333~line 339:

 if (live_mode) {
    decoder->params.AsyncDepth = 1;
    decoder->bs.DataFlag = MFX_BITSTREAM_COMPLETE_FRAME;
    /* Hack for H264 low-latency streaming */
    if (decoder->params.mfx.CodecId == MFX_CODEC_AVC)
      decoder->params.mfx.DecodedOrder = 1;
  }

And remember sender's sps.pic_order_cnt_type ==  2.

So ALL possible solutions have been applied,
but the latency is still there.

Any suggestion?

0 Kudos
2 Replies
Dmitry_E_Intel
Employee
680 Views

I believe the right sequence can be the following:

1) Please submit a bug against https://github.com/intel/gstreamer-media-SDK/

2) If 'mfxh264dec' maintainers confirm it's a MSDK issue, then the issue can be filled against https://github.com/Intel-Media-SDK/MediaSDK

0 Kudos
Mark_L_Intel1
Moderator
680 Views

Hi Dou,

I noticed you are running decoding from RTSP stream behind GStreamer plugin, could you double check it with sample_decode.exe?

To do this, you have to input the raw stream from a file and then set up DepthAsync and DecodeOrder manually by following the example in https://software.intel.com/en-us/forums/intel-media-sdk/topic/704136

Mark

0 Kudos
Reply