DecodeFrameAsync() returns MFX_ERR_MORE_SURFACE when running Intel Media SDK in software implemetati

[Edmond_W]

Dear Sir / Madam,

I 'm using Intel Media SDK 2019R1 and I complied simple_2_decode solution inside media-tutorial-0.0.4 using Visual Studio 2013

I 've some h.264 video data saved from an IP camera and I got different decode behaviour when using -hw and -sw decoding mode of simple_2_decode.

When using the following command line (MFX_IMPL_HARDWARE mode)

simple_decode -hw imsdk_264_dump_000.h264 out_hw.yuv

I can get 8 frames decoded.

But when using the following command line (MFX_IMPL_SOFTWARE mode)

simple_decode -sw imsdk_264_dump_000.h264 out_sw.yuv

The program is stucked with only 1 frame decoded.

Here, I tried to run simple_decode in Visual Studio 2013 debug mode and found that DecodeFrameAsync() will return MFX_ERR_MORE_SURFACE for several frames and finally, the program stucks at session.SyncOperation(syncp, 60000);

So, I would like to consult why in -hw mode, this video data can be decoded, but it is not in -sw mode.

Also, in -sw implemetation, when MFX_ERR_MORE_SURFACE is returned, is the handling in simple_decode code correct or not ?

We need to run Intel Media SDK in software implementation because in some running platform, no GPU would be available.

I 've attached this imsdk_264_dump_000.h264 for your reference.

Thanks and best regards,
Edmond

[JesusE_Intel]

Hi Edmond_W,

Do you see the same behavior with the latest Intel Media SDK 2020 R1 release?

Regards,

Jesus

[Edmond_W]

Hi @JesusE_Intel,

I tried to use the latest Intel Media SDK 2020 R1 release and I still got the same results, i.e. using -hw option, simple_2_decode can proceed decoding this h264 video data file while using -sw option, simple_2_decode will be stucked.

Furthermore, I tried ffmpeg to decode this file using the following command:

ffmpeg -i imsdk_264_dump_000.h264 out_ffmpeg.yuv

I can get this h264 video file decoded by ffmpeg.

So, it seems that only Intel Media SDK (software implementation) has some problem in decoding this h264 video data.

Pls. give me some further advice on this issue.

Thanks,
Edmond

[Mark_L_Intel1]

Hi Edmond,

Let me explain.

The problem you got related to the second frame not being decoded, I believe there is some meta data variety of the input stream. From decoder point of view, the software decoder and hardware decoder does have different behavior to different input streams in some cases.

Anyway, the sample should handle these varieties without problem. The cause is, you are using the tutorial code which is used to show case the API programming, it was not validated against different input streams. It was our fault that we didn't set the correct expectation to the tutorial code which causes confusion.

To check my answer, you can use "sample_decode" with -hw and -sw mode to see if you can decode the same input stream successfully in both mode.

Mark

[Edmond_W]

Hi @Mark_L_Intel1 ,

Thanks for your reply. I tried to use sample_decode to decode the video data file and I still got the same results, i.e. using -hw option, sample_decode can proceed decoding this h264 video data file while using -sw option, sample_decode will be stucked.

I listed the command line output for these 2 cases as follows:

==============================================================================

sample_decode h264 -i imsdk_264_dump_000.h264 -o output_hw.yuv -hw
pretending that aspect ratio is 1:1
Decoding Sample Version 8.4.27.25

Input video AVC
Output format NV12
Input:
Resolution 1920x1088
Crop X,Y,W,H 0,0,1920,1080
Output:
Resolution 1920x1080
Frame rate 25.00
Memory type system
MediaSDK impl hw
MediaSDK version 1.31

Decoding started
Frame number: 8, fps: 72.194, fread_fps: 0.000, fwrite_fps: 279.067
Decoding finished

sample_decode h264 -i imsdk_264_dump_000.h264 -o output_sw.yuv -sw
pretending that aspect ratio is 1:1
Decoding Sample Version 8.4.27.25

Input video AVC
Output format NV12
Input:
Resolution 1920x1088
Crop X,Y,W,H 0,0,1920,1080
Output:
Resolution 1920x1080
Frame rate 25.00
Memory type system
MediaSDK impl sw
MediaSDK version 1.28

Decoding started
Frame number: 0, fps: 0.000, fread_fps: 0.000, fwrite_fps: 0.000
<Programe stucked here !!>

==============================================================================

So, it seems that only Intel Media SDK (software implementation) has some problem in decoding this h264 video data.

Pls. give us some further advice on this issue.

Thanks,
Edmond

[Mark_L_Intel1]

Hi Edmond,

Yes, I can reproduce it in Windows platform. I think the direct reason is, the decoder has some issue to the input stream so it stalled, this leaves our decoding loop waiting the decoder come back until failed on timeout.

I think this is a partly expected result, the reason is, our hardware codec and software codec has different design and don't behave the same.

I will submit this bug and don't expected to be investigated quickly. The software codec was introduced as the backup to the hardware codec when it is not available temporarily. We have move away from this direction, it is not included in our open source release but only in our Windows release. The other reason is, there are a lot of third party codec in the market and open source community.

Does this make sense to you? Let me know your feedback.

Mark

[Edmond_W]

Hi Mark,

Thanks for your investigation and confirmed that there are some design differences between the software and hardware codec.

Hopefully, Intel side can continue to improve the software codec in decoding this video stream and make the software and hardware codec have similar behaviour as much as possible in the next Intel Media SDK release.

Thanks,
Edmond

[Mark_L_Intel1]

Hi Edmond,

Sorry for the late response.

I believe this should be EnableMBQP field, did you try it yet?

I will double check with dev team although I am quite sure this is.

I also check LCU support doesn't have the size limitation.

Mark

[Edmond_W]

Hi Mark,

Thanks for your reply.

I haven't tried to set EnableMBQP field as I don't have any information on how to set it in calling Intel Media SDK API.

Seems this EnableMBQP is a field in mfxExtCodingOption3 struct, so do you 've any further information on how to set this field for h264 decoding ?

Please update us if you 've any findings from your dev team.

Thanks,
Edmond

[Mark_L_Intel1]

Oops, I posted in wrong place. My apology.

I just check with dev team, they don't have answer yet. They just say they can reproduce and no more message yet. Let me remind them.

Mark

[Mark_L_Intel1]

Hi Edmond,

Sorry for the delay, I have some report from dev team. They has a conclusion , this should be a rare case for you. Is this acceptable?

The detailed report:

Specific of the stream is that it has a gap in frame_num (4 just after 0). According to 8.2.5.2 Decoding process for gaps in frame_num, this is valid, if gaps_in_frame_num_value_allowed_flag is on (as it in our case). The SW decoding failed on decoding frame 4 (P-frame), when frame with index 0 from reference picture list used used for decoding: according to current implementation, this is "non-existing" frame 3.

So, for P-frame 4 RefPicList0 contains only "non-existing" frame 3. And according to NOTE 4 of 8.2.5.2, this is "unintentional picture loss" situation.

Mark Liu

[Edmond_W]

Hi Mark,

Thanks for your reply and hopefully, Intel Media SDK (software decoding mode) can handle this rare situation in the coming release.

Thanks,

Edmond

[Mark_L_Intel1]

Conclusion after discussion by dev team: this stream is not standard compliant, and we are trying to resolve reference to non-existing frames as containing zeroes.

I will keep you updated

Mark

[Mark_L_Intel1]

Hi Edmond,

Sorry for the delay response,

Dev team told me: this issue was fixed and will be included in the latest release on GitHub. Could you try it and let me know if it can still be reproduced?

Mark

[Edmond_W]

Hi Mark,

Thanks for your reply, I found the latest Intel® Media SDK for Windows release is still 2020 R1 (Build date: 25 June 2020)

So, for fixing this issue, do I need to get the newer update of Intel Media SDK release or just to test the latest sample_decode from https://github.com/Intel-Media-SDK/MediaSDK would be enough ?

Thanks,
Edmond

[Mark_L_Intel1]

Hi Edmond,

The Windows release is scheduled one time a year, so it will be in the middle of this year.

GitHub release is more frequent, it will have release for each quarter and some other hot-fixes.

For samples, you can use it directly from GitHub. So let me know if there is still an issue and I will ask.

Mark

[Edmond_W]

Hi Mark,

I just pulled the latest code from https://github.com/Intel-Media-SDK/MediaSDK and complied the sample_decode solution at .\msdk\samples\sample_decode\sample_decode.sln using VS 2015 and with Intel Media SDK release 2020 R1.

The issue is still here, i.e.

==================================================================
sample_decode h264 -i imsdk_264_dump_000.h264 -o output_sw.yuv -sw
pretending that aspect ratio is 1:1
Decoding Sample Version 8.4.27.0

Input video AVC
Output format NV12
Input:
Resolution 1920x1088
Crop X,Y,W,H 0,0,1920,1080
Output:
Resolution 1920x1080
Frame rate 25.00
Memory type system
MediaSDK impl sw
MediaSDK version 1.28

Decoding started
Frame number: 0, fps: 0.000, fread_fps: 0.000, fwrite_fps: 0.000
<Programe stucked here !!>
==================================================================

Would you pls. advise me further on this issue.

Thanks.
Edmond

[Mark_L_Intel1]

Hi Edmond,

Thanks for the checking.

It seems like the fix was also impacted in the codec library, so we must wait for the new release, I am double checking with dev team now.

Mark Liu

[Mark_L_Intel1]

Hi Edmond,

Yes, I have confirmed the fix is in the library so you have to wait for the Media SDK Windows release that will be in the middle of this year.

I will keep you updated.

Mark Liu