low latency decode problem

[Guilherme_L_2]

I'm developing a encode/decode lib for a video conference software, it's almost ready with exception of the decode model.

I have been followed the "Video Conferencing features of Intel® Media Software Development Kit" article to set up the parameters of encode and decode in order to improve performace.

But it isn't working well in the decoding side, the colors of the frame starts to mix up with time and in some minutes it turned in to a mess... The only thing i had figured out is always after 11 frames the return from DecodeFrameAsync changes from MFX_ERR_NONE to MFX_WRN_VIDEO_PARAM_CHANGED and it repeats after every 11 frames, my GOP size is 12, so it's obviously connected and it seems to make sense because the frame is different.

The thing i do different from the samples is the mfxVideoParam structure that i fill with out use the DecodeHeader function. The only two fields it's fill and i don't is the CodecLevel and CodecPreset becouse i don't found much information about what to put on it, i tryed some values but the result is a pink decodified frame so i left it with 0 and move on.

Another thing is the workingSurfaces i'm using, the mfxFrameAllocRequest::NumFrameSuggested give me 18 surfaces that will be needed but the decoding process just use the first three ones.

If someone already had a problem like this, please help me.
I will put a ziped sample of a 1920 x 1080 .yuv in attach.

(The green bar in the top of the sample is a MFX_ALIGN16 problem with the 1920 x 1080 resolution and i simply cut it of in exibition.. so i think it's not a problem).

Thank you in advance and sorry for the poor english

Guilherme

[Petter_L_Intel]

Hi Guilherme,

Looking at the provided raw frame data it seems the reason for the corruptions in color and artifacts is due to alignmnet issue you pointed out. Please resolve the alignment issue during encoding so that no green bar is generated in encoded stream.

MFX_WRN_VIDEO_PARAM_CHANGED return value from DecodeFrameAsync is normal. It indicates that a sequence header (e.g. SPS) has been detected ( regardless if parameters have changed or not! ).

Regards,
Petter 

[Guilherme_L_2]

Hi Petter,

Thanks for the answer,

I'm not sure about how to cut off the green bar at the encode side because the original video is a 1920 x 1080 and the Height that i need to pass at mfxVideoParam is 1088, the green bar is put by the mfxlib.

Nevertheless, i think it's not the main problem because i have this problem in the 720p video too.

sorry for the unhappy sample, i'm sending a 720p sample now with more frames,

hope you can help

Regards,

Guilherme

[Petter_L_Intel]

Hi Guilherme,

I do see the artifact now in the raw stream you provided.

Do you see the same artifacts if you decode the stream using Media SDK sample code (sample_decode) and for instance ffmpeg?

If so that would indicate that indicate that the encoded stream is having the artifacts and not a result of the decoding process.

Regarding alignment. Please refer to the Media SDK sample for details on how to resolve that issue. Surface sizes must be of certain aligned size, but the actual frame size does not.

Regards,
Petter

[Guilherme_L_2]

Hi Petter,

In fact, i don't have tested my last version with the sample_decode but i used the simple_decode and the result is perfect.

I didn't used the instance ffmpeg, but in my source code i have a record rotine that is very similar and the output .mp4 file is perfect too.

i made this first test using a recorded elementary h264 file from my encode module and not with a network stream, it will resquest a little more time to be structured. I will do that to test, but i don't think it's the real problem because the transmission works extremely good with ffmpeg as encoder and decoder.

First, i will try feed the decode with a stream file to see what happens.

Every tip is welcome.

Regards,

Guilherme

[Guilherme_L_2]

Hi,

Just to register the problem and the solution.

The frames comes from the network to my application one by one, that way i don't need to read it in blocks of 1024x1024 like the samples do when read the stream from a file. I have the entire encoded frame in a buffer and because of that i was ignoring the mfxBitsrteam::DataOffset and i was just doing something like this:

myMfxBitstream->Data = encodedFrameBufferFromNetwork;

myMfxBitstream->DataLength = sizeOfNetworkBuffer

myMfxBitstream->DataOffset = 0;

the result of it is the artifacts of the sample i've sent.

What i'm doing now is using a memmove and then a memcpy like the samples do with the stream coming from the file.

I still have a delay and i'm trying to optimize the process.

My question is what the DataOffset value means? and if there is any way to do not make that shift and use a different buffer in the mfxBitstream::data in every frame

Thanks in advance and sorry the poor english

Guilherme

[Petter_L_Intel]

Hi Guilherme,

DataOffset indicates how much or the provided bistream that was processed by the decoder.

Note that, if you know that the input bitstream to the decoder contains exactly one frame then you may benegfit from setting the DataFlag = MFX_BITSTREAM_COMPLETE_FRAME

Regards,
Petter 

[Guilherme_L_2]

Hi Petter,

Thanks, i'm already using DataFlag MFX_BITSTREAM_COMPLETE_FRAME.

The DataOffset always have the size of the last frame i had put in the mfxBitstream, so every time i will get a new frame to the decode process i move the data that i recently put on mfxBitstream to the begining of the buffer and then copy the new frame data to the end of the buffer, so i always have two frames in mfxBitstream::data, that's it? or i miss something?

regards,
Guilherme

[j_miles]

A bit side-tracked:

Petter Larsson (Intel) wrote:

MFX_WRN_VIDEO_PARAM_CHANGED return value from DecodeFrameAsync is normal. It indicates that a sequence header (e.g. SPS) has been detected ( regardless if parameters have changed or not! ).

This is not perfectly clear from the documentation, so please consider writing it out more directly (in particular in the section on 'Multiple Sequence Headers' in the latest SDK manual). This is probably made further unclear by the choice of naming of the status code (i.e. containing the word _CHANGED). Repetition of SPS's is allowed by the standard, and such repetitions are quite normal to allow decoding recovery. The behavior that the warning is presented for also unchanged SPS's requires the user to always check if the SPS has actually changed (if they need that kind of info). Is there a completely consistent method for checking if the parameters are unchanged/changed? (e.g. directly comparing the mfxVideoParam structures before and after?)

Thanks

 - Jay

[j_miles]

Hi,

Setting AsyncDepth=1 and flagging MFX_BITSTREAM_COMPLETE_FRAME on data may reduce latency (by removing additional latency) but otherwise it seems to us that the decoder adheres strictly to the max. DPB size. And in my view that is also the right thing from a standards compliance perspective. In that case, the decoder returns MFX_ERR_MORE_DATA successively until the DPB is full and the first (possibly reordered) frame can be output. This is controlled directly by the (possibly inferred) value of 'max_dec_frame_buffering' in the SPS. The only exception to this is if 'pic_order_cnt_type' is equal to 2 (output order = decoding order), where frames are output immediately regardless of the DPB size. Please correct me if I'm wrong, Petter.

It would be nice though with some option of forcing a "low delay mode" for the cases where an application knows that the (possibly inferred) 'max_dec_frame_buffering' value is larger than necessary. Ideally, it could be automatic... 

Thanks

 - Jay

[Petter_L_Intel]

Hi Jay,

your assessment is correct. To ensure overall pipeline low latency you must make sure that your encoder is configured to create a stream with DPB size = 1 as illustrated in the Media SDK sample_videoconf sample.

I agree that the naming of some of the Media SDK status codes are confusion. We will work to enhance the reference manual to improve the description.  There is currently no way to check which parameters were changed so it's up to the application to perform a complete parameter set comparison. This is a known limitation which we will try to address in future releases of the SDK.

Regards,
Petter