Media (Intel® oneAPI Video Processing Library, Intel Media SDK)
Access community support with transcoding, decoding, and encoding in applications using media tools from Intel. This includes Intel® oneAPI Video Processing Library and Intel® Media SDK.
Announcements
Welcome to the Intel Community. If you get an answer you like, please mark it as an Accepted Solution to help others. Thank you!

H264 Decoder latency (again)

Carsten_K_
Beginner
186 Views

Hi,

i have seen many topics on H264 decoder latency in this forum, but my problem is not solved yet.

I feed the decoder with standard H264- IPPPP streams. I try to decode the stream on a picture by picture base (i allways stuff in a complete frame!), but the decoder does NOT return the decoded frame immedeatly. In many cases he expectes "more data".

My decoding parameters are:

  • NO B-Frames!
  • m_mfxBS.DataFlag = MFX_BITSTREAM_COMPLETE_FRAME;
  • m_mfxVideoParams.AsyncDepth = 1;
  •  NumThread = 1 what is not required!

First topic/question/bug:

When i put in the first IDR frame (SPS+PPS+SliceHdr+IDR Frame Data) i expect that DecodeFrameAsync(..)  returns with MFX_NO_ERROR and the decoded frame is available immediately. THERE IS NO REASON (!!!NO!!!!!) that  DecodeFrameAsync returns with MFX_ERR_MORE_DATA!

In an IPPPPP stream should every frame  be avaible immediately after DecodeFrameAsync()

When i decode streams, wich were encoded with the MediaSDK (with  mfxExtCodingOption::MaxDecFrameBuffering = 1), there is no problem, the decoder behaves "normally" and the decoded frame is returned immediately. BUT i have many streams (wich were not Media SDK encoded), where the DecodeFrameAsync returns with MFX_ERR_MORE_DATA. In this case i had to stuff in up to 20 Bytes of the following P-Frame (more than the complete slice header of the following P) to retrieve the decoded IDR. THIS IS NOT NESSARY, and in streaming situations with low fps rates (say 1 fps) unacceptable, because this behavior adds a 1s latency to the decoding path! I cant wait 1s for the next frame ... !!!

I add screenshots of  the SPS+PPS+SIdr-Slice HDR  of the two streams later. One is called Intel.h264 (wich returns MFX_NO_ERROR) the other is called error.h264 ( wich returns MFX_ERR_MORE_DATA). As you can see, the only difference between the streams (except some unimportant vui_flags), is the pic_order_cnt_type.

Second topic/question/bug:

In the stream error.h264 you can see, that the sequence_parameter_set says

  • max_num_ref_frames =1
  • vui_parameters_presesent=1
  • -> bitstream_restriction_flag=1
  • -> -> max_dec_frame_buffering=1

The call  

   // calculate number of surfaces required for decoder
    sts = m_pmfxDEC->QueryIOSurf(&m_mfxVideoParams, &Request);

returns with Request.NumFrameSuggested=10 ! Why???? The SPS said: max_num_ref_frames =1 and max_dec_frame_buffering=1, so NumFramesSuggested should be 2! This results in a big waiste of memory (in multi channel systems) and forces me to parse the headers myself again to retrieve the CORRECT NumFramesSuggested! :(

Do you have any ideas to fix the problems???

If your devellopers need teststreams to fix the BUGS, i can post some ...

Screenshots

Error.h264:

Intel.h264

0 Kudos
5 Replies
Carsten_K_
Beginner
186 Views

There was something wrong with the pictures.. here they are:

Error.h264

Intel.h264

OTorg
New Contributor III
186 Views

Can't see your screenshots. Please, upload Intel.h264 and error.h264 (small portion with SPS, PPS and a few frames is quite enough).

Petter_L_Intel
Employee
186 Views

Hi Carten,

please provide the streams so that we can explore the behavior locally. Please also share the graphics driver version and HW you are using.

Regarding your question about required # of decode surface buffers. This is a known limitation, due to underlying HW behavior (and graphics driver)  a greater number of surface buffers are allocated than is actually needed. There are plans to try to resolve this in future HW platforms and driver drops.

Regards,
Petter 

Oh_i_
Beginner
186 Views

Hi Carten,

I faced the same issues... about the first:

I just made a "fake" call to the decoder, if it replies "more data is needed", it takes no time and I can decode 1 frame at a time, with no latency! (i think it required a new "surface") it returns the decoded frame fully.

sbout the second, i guess it is a bug... usually it is not critical

Oh_i_
Beginner
186 Views

Hi Carten,

I faced the same issues... about the first:

I just made a "fake" call to the decoder, if it rreplies "more data is needed", it takes no time and I can decode 1 frame at a time, with no latency! (i think it required a new "surface", but ut returns the decoded frame fully.

Reply