Media (Intel® oneAPI Video Processing Library, Intel Media SDK)
Access community support with transcoding, decoding, and encoding in applications using media tools from Intel. This includes Intel® oneAPI Video Processing Library and Intel® Media SDK.
Announcements
Welcome to the Intel Community. If you get an answer you like, please mark it as an Accepted Solution to help others. Thank you!

VPP problem with resize and letterbox

Ken_S_
Beginner
143 Views

Hi there,

I wrote my application referencing both simple_5_transcode_opaque_async_vppresize and sample_multi_transcode. It turned out when I do VPP resize and letterboxing, the video is not letterboxed.

The strange part is that when I added the ResizeAndLetterBox routine to sample_multi_transcode, it WORKS! But, when I added the same ResizeAndLetterBox routine to simple_5_transcode_opaque_async_vppresize app, it DOESN'T WORK!

Here are the mfxFrameInfo goint in and out of Decoder, VPP In/Out, Encoder:
[My App/simple_5_transcode_opaque_async_vppresize]

Decode Param: 
      CodecId: 541283905
      Profile: 100
        Level: 31
  TargetUsage: 0
GopSize(flag): 1(0)
   IpInterval: 0
  IdrInterval: 0
  RateControl: 0
    FrameInfo:
         WxH (AR): 1280x720 (1:1)
      CropX,Y,W,H: 0,0,1280,720
        Framerate: 6246/250
            Field: 1
 VPPIn Param: 
         WxH (AR): 1280x720 (1:1)
      CropX,Y,W,H: 0,0,1280,720
        Framerate: 6246/250
            Field: 1
VPPOut Param: 
         WxH (AR): 720x224 (1:1)
      CropX,Y,W,H: 165,0,390,220
        Framerate: 6246/250
            Field: 1
Encoder Param: 
      CodecId: 541283905
      Profile: 66
        Level: 31
  TargetUsage: 4
GopSize(flag): 50(1)
   IpInterval: 1
  IdrInterval: 0
  RateControl: 1
    FrameInfo:
         WxH (AR): 720x224 (0:0)
      CropX,Y,W,H: 0,0,720,220
        Framerate: 6246/250
            Field: 1

 

[sample_multi_transcode]

Decode picture:
    CodeId    541283905
    Profile    100
    Level    31
    TargetUsage:    0
    GopSize(flag):    1(0)
    IpInterval:    0
    IdrInterval:    0
    RateControl:    0
    Resolution (AR)    1280x720 (1:1)
    Crop X,Y,W,H    0,0,1280,720
    Framerate:    6246/250
    Field:    1
VPPIn picture:
    Resolution (AR)    1280x720 (1:1)
    Crop X,Y,W,H    0,0,1280,720
    AspectRatio    1:1
VPPOut picture:
    Resolution (AR)    720x224 (1:1)
    Crop X,Y,W,H    165,0,390,220
    AspectRatio    1:1
Encode picture:
    CodeId    541283905
    Profile    0
    Level    0
    TargetUsage:    4
    GopSize(flag):    0(0)
    IpInterval:    0
    IdrInterval:    0
    RateControl:    1
    Resolution (AR)    720x224 (0:0)
    Crop X,Y,W,H    0,0,720,220
    Framerate:    6246/250
    Field:    1

 

The result are:
[My app/simple_5_transcode_opaque_async_vppresize]

simple_5_transcode_opaque_async_vppresize.png

[sample_multi_transcode]

sample_multi_transcode.png

Both apps are running on the same system, same input file. They both use OpaqueSurface.
I tried to trace down the call stacks on both applications to see if there are any difference when they initialized. The call sequences are nearly identical! At this point, I am pretty much scratching my head off. Any help you can help would greatly appreciated!

[System Info]
OS: SLES12
CPU: Intel XEON E3-1285
MSDK: 2015R3

Input file is h264.

Attached is a modified simple_5_transcode_opaque_async_vppresize that includes the identical ResizeAndLetterBox routine used in both apps (one works and one doesn't). Look for doResizeNBoxVPP() routine.

Many thanks in advance,
Ken S.

0 Kudos
4 Replies
Sravanthi_K_Intel
143 Views

Hi Ken,

In the sample, the output of VPP params are fed into the input of the encode params (crop values, width and height). In the tutorial code that you have, that is not the case(crop values differ between vppOut and EncIn). If you match them up, you will see the letterboxing behavior.

Let me know if that fixes the issue - I was able to achieve the effect by the above-mentioned modification. If you are not, we can dig deeper.

In the sample: MSDK_MEMCPY_VAR(m_mfxEncParams.mfx.FrameInfo, &m_mfxVppParams.vpp.Out, sizeof(mfxFrameInfo));

This is missing in the tutorial.

Ken_S_
Beginner
143 Views

Hi Sravanthi,

Thanks for your quick response. I tried that before. In fact, I tried a lot of different CropX,Y,W,H settings and none of them worked. Unfortunately that won't fix the problem.

This is what I did,
    memcpy(&mfxEncParams.mfx.FrameInfo, &VPPParams.vpp.Out, sizeof(mfxFrameInfo));
    mfxEncParams.mfx.FrameInfo.CropX = 0; // adjust the Crop X, Y coordinate, otherwise, the video gets shifted
    mfxEncParams.mfx.FrameInfo.CropY = 0;

Now the mfxEncParams will have these settings:
Encoder Param: 
      CodecId: 541283905
      Profile: 66
        Level: 31
  TargetUsage: 4
GopSize(flag): 50(1)
   IpInterval: 1
  IdrInterval: 0
  RateControl: 1
    FrameInfo:
         WxH (AR): 720x224 (0:0)
      CropX,Y,W,H: 0,0,390,220

        Framerate: 6246/250
            Field: 1

The result screenshot is as follow:
sample_multi_transcode_2.png

Although it does preserves the aspect ratio of the video, there are significant problems:
1) It doesn't do letterbox (in this case, it doesn't do pillarbox)
2) if you ffprobe the video, the video size is 390x220, NOT 720x220
    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 390x220 [SAR 1:1 DAR 39:22], 760 kb/s, 24.98 fps, 24.98 tbr, 1200k tbn, 49.97 tbc (default)
3) when I played it back with vlc, it gave me a waring:
[h264 @ 0x7f5c94c7cd20] brainfart cropping not supported, this could look slightly wrong ...

Most importantly as I mentioned before, the SAME logic works CORRECTLY with sample_multi_transcode.  One thing I didn't clarify in my earlier post is that sample_multi_transcode itself DOES NOT do resize and letterbox. I need to add the same logic "doResizeNBoxVPP() " to sample_multi_transcode  as well. And, when I did that, it works as desired. Unfortunately, the SAME logic doesn't work in simple_5_transcode_opaque_async_vppresize.

When you say, you get a correct letterbox behavior, can you kindly share the code you modified with me? Perhaps I am making some obvious mistakes.

Thanks,
Ken S.

 

 

Ken_S_
Beginner
143 Views

Hi Sravanthi,

I finally tracked down the problem. It was the Frame Surface on the tutorial sample code. Basically when we created the VppOut FrameSurface, the VppOut FrameInfo structure should be used instead of the EncIn FrameInfo. In most cases, VppOut == EncIn. But, for my scenario (resize and crop), the FrameInfo is different between VppOut and EncIn.


    mfxFrameSurface1** pSurfaces2 = new mfxFrameSurface1 *[nSurfNumVPPEnc];
    MSDK_CHECK_POINTER(pSurfaces2, MFX_ERR_MEMORY_ALLOC);
    for (int i = 0; i < nSurfNumVPPEnc; i++) {
        pSurfaces2 = new mfxFrameSurface1;
        MSDK_CHECK_POINTER(pSurfaces2, MFX_ERR_MEMORY_ALLOC);
        memset(pSurfaces2, 0, sizeof(mfxFrameSurface1));
        //memcpy(&(pSurfaces2->Info), &(EncRequest.Info), sizeof(mfxFrameInfo));
        memcpy(&(pSurfaces2->Info), &(VPPParams.vpp.Out), sizeof(mfxFrameInfo));
    }

After I changed it to use VppOut FrameInfo, everything works as expected.


Thanks,
Ken S.

Sravanthi_K_Intel
143 Views

Thanks for updating the post Ken! Glad you got it working. When using a media pipeline (dec-vpp-enc), using the correct parameters is probably the tricky part! Thanks again.

Reply