H264 decoder latency by TopoEdit

[Lily_D_]

Hi

I have downloaded the latest Intel SDK (Media SDK 2012 R3), and registered the mfx_mft_h264vd_32.dll to my computer (Win8, 9200, 32bit).

By using the TopoEdit (Media Foundation tool), I add the video capture source, add the Intel Media SDK H.264 Decoder MFT, add the EVR, and then preview. I find that it previewed successfully but with some latency.

I debugged the source code(sample_mfoundation_plugins) attaching to the TopoEdit.exe. According to the trace information, at the beginning, the decoder would buffer some samples before decoding and calling ProcessOutput to output sample.

Later, I set

 mfxVideoParam::AsyncDepth = 1 and

mfxBitStream::DataFlag = MFX_BITSTREAM_COMPLETE_FRAME

The trace information show that the decoder would buffer 5 samples before decoding. The format of the video from camera capture is H264, 1280*720, 10Fps.

However I want to preview the H264 real-time without latency.

Who can tell me how to implement the real-time preview H264?

Or which parameters should be set to control that the decoder will not buffer samples before decoding.

Thanks!

[celli4]

Lily, This is an interesting question, I am a developer working with the Media SDK too. Although I have done little with Media Foundation and the Media SDK, I have done some work with DirectShow and the Media SDK. If you are indeed working with a USB H.264 camera under Media Foundation, I think you will find fair bit of latency even with little latency in the H.264 decoder. I also believe the rendering filter at the end of your graph may add a fair bit of latency in order to have a playback buffer capable of smoothing out jitter in the frame arrivals, you may want to look into this also. Are you able to share your Topo graph? Cameron

[Lily_D_]

Dear Cameron: The Topo graph just like this. I also test the latency with the format H264, 640*480, 30Fps. The trace information shows that the decoder would buffer 16 samples before decoding. The latency of 1280*720, 10Fps is about 720ms. Today the latency of 640*480 30Fps has not been tested due to some reason, but in my opinion, it may 700ms~800ms. So, since the fair bit of latency is not avoided, I just want to know how to make the decoder not buffer samples or buffer one sample only to reduce the latency as far as possible. Thank you Lily.

camkego wrote:

Lily,

This is an interesting question, I am a developer working with the Media SDK too. Although I have done little with Media Foundation and the Media SDK, I have done some work with DirectShow and the Media SDK.

If you are indeed working with a USB H.264 camera under Media Foundation,
I think you will find fair bit of latency even with little latency in the H.264 decoder.
I also believe the rendering filter at the end of your graph may add a fair bit of latency in order to have a playback buffer
capable of smoothing out jitter in the frame arrivals, you may want to look into this also.

Are you able to share your Topo graph?

Cameron

[Lily_D_]

Topo Graph

[celli4]

I suspect you will have a hard time reducing the latency much: - you are using a USB Camera probably will little concern for low-latency design - the camera most likely cannot generate an Intel MSDK conforming low-latency AVC stream - probably the Media Foundation renderer does buffering to ensure it can deliver frames at the right time - there may be other buffer points in the MF graph which you have little or no control of I wouldn't be suprised if you figure out how to reduce some latency, it just might be hard or impossible to break 300-250ms, at least this has been my experience with 3rd party filters and DirectShow. Please let us know here about your progress, I only have suspicions, not enough hard experience with MF to say for sure. Hopefully I am wrong. Regards, Cameron

[Petter_L_Intel]

Hi, Regarding configuring Intel Media SDK components for low latency and to enable other features related to video conferencing, please refer to the following white paper: http://software.intel.com/en-us/articles/video-conferencing-features-of-intel-media-software-development-kit To achieve low latency decode using Media SDK you are also impacted by the format of the input stream. For instance, if the input stream uses B-frames then additional decoder latency will naturally occur. Regards, Petter

[Lily_D_]

Hi Petter: In my test, the input stream use 9 B-frames between two I-frame. And then I made the input stream using 2 B-frames between two I-frame or using totally I-frames, but the latency do not improve. The trace information also show that the decoder will buffer 5 samples before decoding. Is there an internal buffer set to cache data in the decoder? If it has, how can i change the internal buffer size? Best Regards Lily

Petter Larsson (Intel) wrote:

Hi,

Regarding configuring Intel Media SDK components for low latency and to enable other features related to video conferencing, please refer to the following white paper:
http://software.intel.com/en-us/articles/video-conferencing-features-of-...

To achieve low latency decode using Media SDK you are also impacted by the format of the input stream. For instance, if the input stream uses B-frames then additional decoder latency will naturally occur.

Regards,
Petter

[Petter_L_Intel]

Hi Lily, if you look at the referenced white paper you will find information on how to encode your stream so that decoder can decode with low latency. Having no B-frames is essential, but also make sure that decoder picture buffer (DPD) gets set to 1 as described in the paper. Regards, Petter

[Lily_D_]

I have resloved the problem by using DecodeFrameAsync(NULL,..) to pull the cached sample. Thanks for everyone's help.