Media (Intel® Video Processing Library, Intel Media SDK)
Access community support with transcoding, decoding, and encoding in applications using media tools like Intel® oneAPI Video Processing Library and Intel® Media SDK
Announcements
The Intel Media SDK project is no longer active. For continued support and access to new features, Intel Media SDK users are encouraged to read the transition guide on upgrading from Intel® Media SDK to Intel® Video Processing Library (VPL), and to move to VPL as soon as possible.
For more information, see the VPL website.
3056 Discussions

DecodeFrameAsync() returns MFX_ERR_MORE_SURFACE when running Intel Media SDK in software implemetati

Edmond_W
Novice
2,757 Views

Dear Sir / Madam,

I 'm using Intel Media SDK 2019R1 and I complied simple_2_decode solution inside media-tutorial-0.0.4 using Visual Studio 2013

I 've some h.264 video data saved from an IP camera and I got different decode behaviour when using -hw and -sw decoding mode of simple_2_decode.

When using the following command line (MFX_IMPL_HARDWARE mode)

simple_decode -hw imsdk_264_dump_000.h264 out_hw.yuv

I can get 8 frames decoded.

But when using the following command line (MFX_IMPL_SOFTWARE mode)

simple_decode -sw imsdk_264_dump_000.h264 out_sw.yuv

The program is stucked with only 1 frame decoded.

Here, I tried to run simple_decode in Visual Studio 2013 debug mode and found that DecodeFrameAsync() will return MFX_ERR_MORE_SURFACE for several frames and finally, the program stucks at session.SyncOperation(syncp, 60000);

So, I would like to consult why in -hw mode, this video data can be decoded, but it is not in -sw mode.

Also, in -sw implemetation, when MFX_ERR_MORE_SURFACE is returned, is the handling in simple_decode code correct or not ?

We need to run Intel Media SDK in software implementation because in some running platform, no GPU would be available.

I 've attached this imsdk_264_dump_000.h264 for your reference.

Thanks and best regards,
Edmond

 

0 Kudos
19 Replies
JesusE_Intel
Moderator
2,728 Views

Hi Edmond_W,


Do you see the same behavior with the latest Intel Media SDK 2020 R1 release?


Regards,

Jesus


0 Kudos
Edmond_W
Novice
2,720 Views

Hi @JesusE_Intel,

I tried to use the latest Intel Media SDK 2020 R1 release and I still got the same results, i.e. using -hw option, simple_2_decode can proceed decoding this h264 video data file while using -sw option, simple_2_decode will be stucked.

Furthermore, I tried ffmpeg to decode this file using the following command:

ffmpeg -i imsdk_264_dump_000.h264 out_ffmpeg.yuv

I can get this h264 video file decoded by ffmpeg.

So, it seems that only Intel Media SDK (software implementation) has some problem in decoding this h264 video data.

Pls. give me some further advice on this issue.

Thanks,
Edmond

0 Kudos
Mark_L_Intel1
Moderator
2,664 Views

Hi Edmond,

 

Let me explain.

 

The problem you got related to the second frame not being decoded, I believe there is some meta data variety of the input stream. From decoder point of view, the software decoder and hardware decoder does have different behavior to different input streams in some cases.

 

Anyway, the sample should handle these varieties without problem. The cause is, you are using the tutorial code which is used to show case the API programming, it was not validated against different input streams. It was our fault that we didn't set the correct expectation to the tutorial code which causes confusion.

 

To check my answer, you can use "sample_decode" with -hw and -sw mode to see if you can decode the same input stream successfully in both mode.

 

Mark

 

0 Kudos
Edmond_W
Novice
2,658 Views

Hi @Mark_L_Intel1 ,

Thanks for your reply. I tried to use sample_decode to decode the video data file and I still got the same results, i.e. using -hw option, sample_decode can proceed decoding this h264 video data file while using -sw option, sample_decode will be stucked.

I listed the command line output for these 2 cases as follows:

==============================================================================

sample_decode h264 -i imsdk_264_dump_000.h264 -o output_hw.yuv -hw
pretending that aspect ratio is 1:1
Decoding Sample Version 8.4.27.25


Input video AVC
Output format NV12
Input:
Resolution 1920x1088
Crop X,Y,W,H 0,0,1920,1080
Output:
Resolution 1920x1080
Frame rate 25.00
Memory type system
MediaSDK impl hw
MediaSDK version 1.31

Decoding started
Frame number: 8, fps: 72.194, fread_fps: 0.000, fwrite_fps: 279.067
Decoding finished


sample_decode h264 -i imsdk_264_dump_000.h264 -o output_sw.yuv -sw
pretending that aspect ratio is 1:1
Decoding Sample Version 8.4.27.25


Input video AVC
Output format NV12
Input:
Resolution 1920x1088
Crop X,Y,W,H 0,0,1920,1080
Output:
Resolution 1920x1080
Frame rate 25.00
Memory type system
MediaSDK impl sw
MediaSDK version 1.28

Decoding started
Frame number: 0, fps: 0.000, fread_fps: 0.000, fwrite_fps: 0.000
<Programe stucked here !!>

==============================================================================

So, it seems that only Intel Media SDK (software implementation) has some problem in decoding this h264 video data.

Pls. give us some further advice on this issue.

Thanks,
Edmond

 

0 Kudos
Mark_L_Intel1
Moderator
2,612 Views

Hi Edmond,


Yes, I can reproduce it in Windows platform. I think the direct reason is, the decoder has some issue to the input stream so it stalled, this leaves our decoding loop waiting the decoder come back until failed on timeout.


I think this is a partly expected result, the reason is, our hardware codec and software codec has different design and don't behave the same.


I will submit this bug and don't expected to be investigated quickly. The software codec was introduced as the backup to the hardware codec when it is not available temporarily. We have move away from this direction, it is not included in our open source release but only in our Windows release. The other reason is, there are a lot of third party codec in the market and open source community.


Does this make sense to you? Let me know your feedback.


Mark


0 Kudos
Edmond_W
Novice
2,584 Views

Hi Mark,

Thanks for your investigation and confirmed that there are some design differences between the software and hardware codec.

Hopefully, Intel side can continue to improve the software codec in decoding this video stream and make the software and hardware codec have similar behaviour as much as possible in the next Intel Media SDK release.

Thanks,
Edmond

0 Kudos
Mark_L_Intel1
Moderator
2,511 Views

Hi Edmond,


Sorry for the late response.


I believe this should be EnableMBQP field, did you try it yet?


I will double check with dev team although I am quite sure this is.


I also check LCU support doesn't have the size limitation.


Mark


0 Kudos
Edmond_W
Novice
2,502 Views

Hi Mark,

Thanks for your reply.

I haven't tried to set EnableMBQP field as I don't have any information on how to set it in calling Intel Media SDK API.

Seems this EnableMBQP is a field in mfxExtCodingOption3 struct, so do you 've any further information on how to set this field for h264 decoding ?

Please update us if you 've any findings from your dev team.

Thanks,
Edmond

0 Kudos
Mark_L_Intel1
Moderator
2,488 Views

Oops, I posted in wrong place. My apology.

 

I just check with dev team, they don't have answer yet. They just say they can reproduce and no more message yet. Let me remind them.

 

Mark

 

0 Kudos
Mark_L_Intel1
Moderator
2,435 Views

Hi Edmond,


Sorry for the delay, I have some report from dev team. They has a conclusion , this should be a rare case for you. Is this acceptable?


The detailed report:

Specific of the stream is that it has a gap in frame_num (4 just after 0). According to 8.2.5.2 Decoding process for gaps in frame_num, this is valid, if gaps_in_frame_num_value_allowed_flag is on (as it in our case). The SW decoding failed on decoding frame 4 (P-frame), when frame with index 0 from reference picture list used used for decoding: according to current implementation, this is "non-existing" frame 3.


So, for P-frame 4 RefPicList0 contains only "non-existing" frame 3. And according to NOTE 4 of 8.2.5.2, this is "unintentional picture loss" situation.


Mark Liu


0 Kudos
Edmond_W
Novice
2,343 Views

Hi Mark,

Thanks for your reply and hopefully, Intel Media SDK (software decoding mode) can handle this rare situation in the coming release.

Thanks,

Edmond

0 Kudos
Mark_L_Intel1
Moderator
2,301 Views

Conclusion after discussion by dev team: this stream is not standard compliant, and we are trying to resolve reference to non-existing frames as containing zeroes.


I will keep you updated


Mark


0 Kudos
Mark_L_Intel1
Moderator
2,129 Views

Hi Edmond,


Sorry for the delay response,


Dev team told me: this issue was fixed and will be included in the latest release on GitHub. Could you try it and let me know if it can still be reproduced?


Mark


0 Kudos
Edmond_W
Novice
2,125 Views

Hi Mark,

Thanks for your reply, I found the latest Intel® Media SDK for Windows release is still 2020 R1 (Build date: 25 June 2020)

So, for fixing this issue, do I need to get the newer update of Intel Media SDK release or just to test the latest sample_decode from https://github.com/Intel-Media-SDK/MediaSDK would be enough ?

Thanks,
Edmond

0 Kudos
Mark_L_Intel1
Moderator
2,120 Views

Hi Edmond,


The Windows release is scheduled one time a year, so it will be in the middle of this year.


GitHub release is more frequent, it will have release for each quarter and some other hot-fixes.


For samples, you can use it directly from GitHub. So let me know if there is still an issue and I will ask.


Mark


0 Kudos
Edmond_W
Novice
2,095 Views

Hi Mark,

I just pulled the latest code from https://github.com/Intel-Media-SDK/MediaSDK and complied the sample_decode solution at .\msdk\samples\sample_decode\sample_decode.sln using VS 2015 and with Intel Media SDK release 2020 R1.

The issue is still here, i.e.


==================================================================
sample_decode h264 -i imsdk_264_dump_000.h264 -o output_sw.yuv -sw
pretending that aspect ratio is 1:1
Decoding Sample Version 8.4.27.0


Input video AVC
Output format NV12
Input:
Resolution 1920x1088
Crop X,Y,W,H 0,0,1920,1080
Output:
Resolution 1920x1080
Frame rate 25.00
Memory type system
MediaSDK impl sw
MediaSDK version 1.28

Decoding started
Frame number: 0, fps: 0.000, fread_fps: 0.000, fwrite_fps: 0.000
<Programe stucked here !!>
==================================================================

Would you pls. advise me further on this issue.

Thanks.
Edmond

0 Kudos
Mark_L_Intel1
Moderator
2,058 Views

Hi Edmond,


Thanks for the checking.


It seems like the fix was also impacted in the codec library, so we must wait for the new release, I am double checking with dev team now.


Mark Liu


0 Kudos
Mark_L_Intel1
Moderator
2,055 Views

Hi Edmond,


Yes, I have confirmed the fix is in the library so you have to wait for the Media SDK Windows release that will be in the middle of this year.


I will keep you updated.


Mark Liu


0 Kudos
Pamela_H_Intel
Moderator
895 Views

Apologies for the delays. Media SDK functionality has been superseded by oneVPL, FFmpeg and GStreamer.


For more information see the oneVPL Landing Page: https://www.intel.com/content/www/us/en/developer/tools/oneapi/onevpl.html where you will find, among other information:


oneVPL version 2.x is an API that is the successor to Intel® Media SDK. The Intel Media SDK project is no longer active and will not be updated. Packages and downloads are still available on GitHub*. For continued support and access to new features, Intel Media SDK users are encouraged to move to oneVPL as soon as possible.

·       Upgrade from Intel Media SDK to oneVPL

·       Intel Media SDK Discontinuation Notice

·       Download from GitHub


Let us know if you have any questions about oneVPL or how to convert from MediaSDK.


0 Kudos
Reply