we have a working H264 Media SDK based media decoder. We've tested it for some "extreme" scenarios like 60 fps Full HD video or 4K H264 streams. Performance and latency is perfect, thanks for the great SDK. Now we've come across a problem with H264 aspect ratio correction. Some H264 streams specify in SPS/PPS, that the aspect ratio after decoding should be changed. When decoding such streams, we can see that the SDK does a good job in DecodeHeader. The mfxVideoParam.mfx.FrameInfo Width and Height members are filled with the correct physical pixel size and the AspectRatioW and AspectRatioH members are filled with the correct aspect ratio correction values as they appear in SPS/PPS. But then QueryIOSurf suggests to allocate a buffer with the physical pixel size (without aspect ratio correction) and the decoding process also creates a surface without the correction, I would expect, however, that the decoder handles the aspect ratio correction itself as it is supposed to do when it sees such information in the bitstream.
Could you please advise me how to correctly deal with streams that use aspect ratio correction? Is there a way to force the decoder to take the aspect ratio correction into consideration during decoding?
Thank you for the feedback of the SDK. We are glad that you are happy with the performance and latency of the SDK.
I am bit confused with your question, as far as I have understood aspect ratio correction is happening in this function
void ConfigureAspectRatioConversion(mfxInfoVPP* pVppInfo) in sample_utils.cpp Ln1589 in sample_decode solution
Since aspect ratio conversion is a VPP function here, so it can be configured after we decode the frame and change the aspect later. Due to which QueryIOSurf is suggesting allocate buffer with the physical size and not with aspect ratio converted.
You might wanna look into the tutorials simple_6_decode_vpp_postproc which can be downloaded from here under tutorial section v0.3 which explain decode+vpp better.
Let me know if I haven't got your question right and It would be great if you can explain with the reference to the existing code from samples/tutorials. By that way we will be looking into the same thing.
For my own understanding can you please explain that what difference in behavior you are seeing if decoder doesn't handle aspect ratio conversion ?
ok to make this clear. I have an H264 stream with physical pixel resolution 704x288 from a XY camera. The camera indicates that the aspect ratio of the stream must be changed after decoding. It does this by including aspect ratio correction information into SPS/PPS, this information is correctly parsed by Media SDK (FrameInfo.Width = 704, FrameInfo.Height = 288, FrameInfo.AspectRatioW = 1, FrameInfo.AspecRatioH = 2). Therefore, the stream must be displayed at 704x576 resolution. Other H264 decoders we've tested in the past handle the aspect ratio correction automatically (for example the Intel IPP decoder), so they ouput 704x576 buffers in such a case. Using the Media SDK decoder, however, we get 704x288 buffers with decoded data.
So, if I understand you correctly, this is standard behaviour and we have to always manually add a VPP to the pipeline to handle a situation when AspectRatioW != AspectRatioH ?
For any type of resizing, we have to include VPP scaling operation in it. Please check simple_decode_vpp_postproc tutorial as I mentioned before. I have tested it by giving an input of 704x288 resolution h264 input and o/p the yuv of 704x576 resolution, which you are doing by changing the o/p and crop width and height.
In decode sample we parse the parameters from the input file so the o/p size is 704x288 unless you want to change it which can be done only if you give o/p params which needs to include VPP in the pipeline.
I am not sure if I have got you correctly about the aspectratioW and aspectratioH here and how you are getting the ratio unless you have changed the o/p params in the code somewhere. Can you please explain how are you giving the desired o/p size?
thanks again for your reply. Unfortunately, I don't know how I could be even more precise in describing the problem. Well, there are H264 streams which do require an aspect ratio correction before being displayed and this is indicated in SPS/PPS. In these cases H264 decoders (mostly) output aspect ratio corrected buffers with decoded data. The Media SDK decoder seems to output buffers without the correction and one has to use VPP to get the corrected result. This is how I finally understand how the Media SDK works and I think that your replies confirm this. The small drawback I see here is that even in a minimalistic "real world" decoder, one has *always* to include the VPP to allow for the possibility of aspect ratio correction. That's the whole story :).
Thanks and have a nice day.