Low Latency Decode of H.264

[simerly__tim]

Dear Tech Support:

I am using the latest version of the Media SDK and trying to decode an H.264 bitstream and need it to provide me with a low latency result.  Hence when I issue the decode call, I need it to provide me back a decoded frame ASAP and not queue up a bunch of them prior to providing me back a surface to draw into.  Currently when I make the following call to decode my bitstream:

sts = m_pmfxDEC->DecodeFrameAsync(pBitstream, &(m_pCurrentFreeSurface->frame), &pOutSurface, &(m_pCurrentFreeOutputSurface->syncp));

it returns me back 16 consecutive MFX_ERR_MORE_SURFACE results until I get a MFX_ERR_NONE result for me to draw into the output surface via a valid non-NULL pOutSurface pointer.  This creates a 16 frame delay and since I'm running in real-time, that creates over 1/2 second of delay for normal 30 FPS video - not cool. 

I have the following during my initialization:

AsyncDepth = 1
bUseHWLib = 1
m_b_DecOutSystem = SYSTEM_MEMORY
memType = D3D9_MEMORY

and the bitstream data flag set for MFX_BITSTREAM_COMPLETE_FRAME

I am providing the decoder with an H.264 bitstream with IP only frames - no B Frames and the first frame I am providing it with is always an I frame.  Can you tell me what I need to do in order for the H.264 decoder to provide me back an MFX_ERR_NONE and a non-null pOutSurface for me to use (hopefully after the first or no later that the 2nd decoder call) instead of having to wait for 16 frames before I can do so?  Please advise as to what I am doing wrong.

Thanks,
Tim

[Mark_L_Intel1]

Hi Tim,

In your configuration argument, I am seeing these: "m_b_DecOutSystem = SYSTEM_MEMORY" and "memType = D3D9_MEMORY"; I don't know what the "SYSTEM_MEMORY" means to you, but if you assigned to MSDK argument with MFX_IOPATTERN_SYSTEM_MEMORY, while D3D9 implies the video memory, you are probably in a slow path.

For a quick check, could you try our sample_decode binary with the same stream to see if it has a long latency? You can use the following arguments:

"-low_latency", "-async 1"; while doing this, you can use argument "-calc_latency" to get the log of the latency.

You can check the following document for the detailed usage of the sample_decode:

https://github.com/Intel-Media-SDK/samples/tree/master/samples/sample_decode

Mark

[simerly__tim]

Hey Mark:

Yes, I have tried running the sample decode in the Media SDK and running into the same problem.  I noticed that I mistyped one of the entries.  I should have wrote:

m_bDecOutSystem = false
m_mfxVideoParams.IOPattern =  MFX_IOPATTERN_OUT_VIDEO_MEMORY

so sorry about the confusion.  I reran the sample decode example and confirmed that it is running with these set as above.  I ran the sample decode example from the command line with the following command line:

sample_decode.exe h264 -i decode_bitstream.264 -r -hw -d3d -ascync 1 -calc_latency > log_low_latency.log

see attached file for the resulting output.  It all looks ok, but when I run through the example, it takes 16 times through the decode call below:

sts = m_pmfxDEC->DecodeFrameAsync( pBitstream, &(m_pCurrentFreeSurface->frame), &pOutSurface, &(m_pCurrentFreeOutputSurface->syncp));

before I get a non-null pOutSurface for me to render into.  Hence the display of the bitstream requires the decoder to decode 16 frames before I can get the 1st frame to render from this decode call above.  Since the first frame that I am sending it is an I frame and all the following frames are P frames, what is going on with the H.264 decoder that is requiring it to decode 16 frames before giving me a frame back that I can render?  Again - I checked on the sample decoder that you provided and single stepping through the program using Visual Studio and it is doing the same thing.  Please advise as to what I may be doing wrong.

Thanks,
Tim
 

[simerly__tim]

Hey Mark:

One more issue, I reran the sample decoder with the same options above but using a "-f 30" option to have the decoder run at 30 FPS, and got a different result from the log - see attached for the updated log file from running with this additional parameter - full command line is below for all 4 cases:

log_low_latency.log
sample_decode.exe h264 -i decode_bitstream.264 -r -hw -d3d -async 1 -low_latency -calc_latency > log_low_latency.log

log_no_low_latency.log
sample_decode.exe h264 -i decode_bitstream.264 -r -hw -d3d -async 1 -calc_latency > log_no_low_latency.log

log_low_latency_30fps.log
sample_decode.exe h264 -i decode_bitstream.264 -r -f 30 -hw -d3d -async 1 -low_latency -calc_latency > log_low_latency_30fps.log

log_no_low_latency_30fps.log
sample_decode.exe h264 -i decode_bitstream.264 -r -f 30 -hw -d3d -async -calc_latency > log_low_latency_30fps.log

Notice from my previous post that I didn't have the -low_latency option selected.  So I am provided all 4 cases above and in the attached file.  Please review the attached logs and let me know if see what may be wrong or you need me to run any other options.

Thanks,
Tim

 

[Dmitry_E_Intel]

Hi Tim,

Please look at this thread:

https://software.intel.com/en-us/forums/intel-media-sdk/topic/704136

Regards,

Dmitry

[simerly__tim]

Hey Dmitry:

Thanks - that was helpful, I had reviewed that earlier, but wasn't sure if it applied since I had a 16 frame delay and I wasn't sure what was meant by changing the DecodeOrder.  Upon further investigation, I see that if I set the flag DecodedOrder in the mfx parameters that it fixes it.  More specifically, per the example sample decoder code, performing this operation at the end of the InitMfxParams(...) call:

m_mfxVideoParams.mfx.DecodedOrder = TRUE;

fixes the problem.  Thanks a lot for your help in getting me pointed in the right direction on allowing me to display the more recently decoded H.264 frame.

Thanks,
Tim