Media (Intel® Video Processing Library, Intel Media SDK)
Access community support with transcoding, decoding, and encoding in applications using media tools like Intel® oneAPI Video Processing Library and Intel® Media SDK
Announcements
The Intel Media SDK project is no longer active. For continued support and access to new features, Intel Media SDK users are encouraged to read the transition guide on upgrading from Intel® Media SDK to Intel® Video Processing Library (VPL), and to move to VPL as soon as possible.
For more information, see the VPL website.

H264 decoder latency by TopoEdit

Lily_D_
Beginner
604 Views

Hi

I have downloaded the latest Intel SDK (Media SDK 2012 R3), and registered the mfx_mft_h264vd_32.dll to my computer (Win8, 9200, 32bit).

By using the TopoEdit (Media Foundation tool), I add the video capture source, add the Intel Media SDK H.264 Decoder MFT, add the EVR, and then preview. I find that it previewed successfully but with some latency.

I debugged the source code(sample_mfoundation_plugins) attaching to the TopoEdit.exe. According to the trace information, at the beginning, the decoder would buffer some samples before decoding and calling ProcessOutput to output sample.

Later, I set

 mfxVideoParam::AsyncDepth = 1 and

mfxBitStream::DataFlag = MFX_BITSTREAM_COMPLETE_FRAME

The trace information show that the decoder would buffer 5 samples before decoding. The format of the video from camera capture is H264, 1280*720, 10Fps.

However I want to preview the H264 real-time without latency.

Who can tell me how to implement the real-time preview H264?

Or which parameters should be set to control that the decoder will not buffer samples before decoding.

Thanks!

0 Kudos
8 Replies
celli4
New Contributor I
604 Views
Lily, This is an interesting question, I am a developer working with the Media SDK too. Although I have done little with Media Foundation and the Media SDK, I have done some work with DirectShow and the Media SDK. If you are indeed working with a USB H.264 camera under Media Foundation, I think you will find fair bit of latency even with little latency in the H.264 decoder. I also believe the rendering filter at the end of your graph may add a fair bit of latency in order to have a playback buffer capable of smoothing out jitter in the frame arrivals, you may want to look into this also. Are you able to share your Topo graph? Cameron
0 Kudos
Lily_D_
Beginner
604 Views
Dear Cameron: The Topo graph just like this. I also test the latency with the format H264, 640*480, 30Fps. The trace information shows that the decoder would buffer 16 samples before decoding. The latency of 1280*720, 10Fps is about 720ms. Today the latency of 640*480 30Fps has not been tested due to some reason, but in my opinion, it may 700ms~800ms. So, since the fair bit of latency is not avoided, I just want to know how to make the decoder not buffer samples or buffer one sample only to reduce the latency as far as possible. Thank you Lily.
camkego wrote:

Lily,

This is an interesting question, I am a developer working with the Media SDK too. Although I have done little with Media Foundation and the Media SDK, I have done some work with DirectShow and the Media SDK.

If you are indeed working with a USB H.264 camera under Media Foundation,
I think you will find fair bit of latency even with little latency in the H.264 decoder.
I also believe the rendering filter at the end of your graph may add a fair bit of latency in order to have a playback buffer
capable of smoothing out jitter in the frame arrivals, you may want to look into this also.

Are you able to share your Topo graph?

Cameron

0 Kudos
Lily_D_
Beginner
604 Views
Topo Graph
0 Kudos
celli4
New Contributor I
604 Views
I suspect you will have a hard time reducing the latency much: - you are using a USB Camera probably will little concern for low-latency design - the camera most likely cannot generate an Intel MSDK conforming low-latency AVC stream - probably the Media Foundation renderer does buffering to ensure it can deliver frames at the right time - there may be other buffer points in the MF graph which you have little or no control of I wouldn't be suprised if you figure out how to reduce some latency, it just might be hard or impossible to break 300-250ms, at least this has been my experience with 3rd party filters and DirectShow. Please let us know here about your progress, I only have suspicions, not enough hard experience with MF to say for sure. Hopefully I am wrong. Regards, Cameron
0 Kudos
Petter_L_Intel
Employee
604 Views
Hi, Regarding configuring Intel Media SDK components for low latency and to enable other features related to video conferencing, please refer to the following white paper: http://software.intel.com/en-us/articles/video-conferencing-features-of-intel-media-software-development-kit To achieve low latency decode using Media SDK you are also impacted by the format of the input stream. For instance, if the input stream uses B-frames then additional decoder latency will naturally occur. Regards, Petter
0 Kudos
Lily_D_
Beginner
604 Views
Hi Petter: In my test, the input stream use 9 B-frames between two I-frame. And then I made the input stream using 2 B-frames between two I-frame or using totally I-frames, but the latency do not improve. The trace information also show that the decoder will buffer 5 samples before decoding. Is there an internal buffer set to cache data in the decoder? If it has, how can i change the internal buffer size? Best Regards Lily
Petter Larsson (Intel) wrote:

Hi,

Regarding configuring Intel Media SDK components for low latency and to enable other features related to video conferencing, please refer to the following white paper:
http://software.intel.com/en-us/articles/video-conferencing-features-of-...

To achieve low latency decode using Media SDK you are also impacted by the format of the input stream. For instance, if the input stream uses B-frames then additional decoder latency will naturally occur.

Regards,
Petter

0 Kudos
Petter_L_Intel
Employee
604 Views
Hi Lily, if you look at the referenced white paper you will find information on how to encode your stream so that decoder can decode with low latency. Having no B-frames is essential, but also make sure that decoder picture buffer (DPD) gets set to 1 as described in the paper. Regards, Petter
0 Kudos
Lily_D_
Beginner
604 Views
I have resloved the problem by using DecodeFrameAsync(NULL,..) to pull the cached sample. Thanks for everyone's help.
0 Kudos
Reply