Community
cancel
Showing results for 
Search instead for 
Did you mean: 
Highlighted
Beginner
37 Views

VPP problem with resize and letterbox

Hi there,

I wrote my application referencing both simple_5_transcode_opaque_async_vppresize and sample_multi_transcode. It turned out when I do VPP resize and letterboxing, the video is not letterboxed.

The strange part is that when I added the ResizeAndLetterBox routine to sample_multi_transcode, it WORKS! But, when I added the same ResizeAndLetterBox routine to simple_5_transcode_opaque_async_vppresize app, it DOESN'T WORK!

Here are the mfxFrameInfo goint in and out of Decoder, VPP In/Out, Encoder:
[My App/simple_5_transcode_opaque_async_vppresize]

Decode Param: 
      CodecId: 541283905
      Profile: 100
        Level: 31
  TargetUsage: 0
GopSize(flag): 1(0)
   IpInterval: 0
  IdrInterval: 0
  RateControl: 0
    FrameInfo:
         WxH (AR): 1280x720 (1:1)
      CropX,Y,W,H: 0,0,1280,720
        Framerate: 6246/250
            Field: 1
 VPPIn Param: 
         WxH (AR): 1280x720 (1:1)
      CropX,Y,W,H: 0,0,1280,720
        Framerate: 6246/250
            Field: 1
VPPOut Param: 
         WxH (AR): 720x224 (1:1)
      CropX,Y,W,H: 165,0,390,220
        Framerate: 6246/250
            Field: 1
Encoder Param: 
      CodecId: 541283905
      Profile: 66
        Level: 31
  TargetUsage: 4
GopSize(flag): 50(1)
   IpInterval: 1
  IdrInterval: 0
  RateControl: 1
    FrameInfo:
         WxH (AR): 720x224 (0:0)
      CropX,Y,W,H: 0,0,720,220
        Framerate: 6246/250
            Field: 1

 

[sample_multi_transcode]

Decode picture:
    CodeId    541283905
    Profile    100
    Level    31
    TargetUsage:    0
    GopSize(flag):    1(0)
    IpInterval:    0
    IdrInterval:    0
    RateControl:    0
    Resolution (AR)    1280x720 (1:1)
    Crop X,Y,W,H    0,0,1280,720
    Framerate:    6246/250
    Field:    1
VPPIn picture:
    Resolution (AR)    1280x720 (1:1)
    Crop X,Y,W,H    0,0,1280,720
    AspectRatio    1:1
VPPOut picture:
    Resolution (AR)    720x224 (1:1)
    Crop X,Y,W,H    165,0,390,220
    AspectRatio    1:1
Encode picture:
    CodeId    541283905
    Profile    0
    Level    0
    TargetUsage:    4
    GopSize(flag):    0(0)
    IpInterval:    0
    IdrInterval:    0
    RateControl:    1
    Resolution (AR)    720x224 (0:0)
    Crop X,Y,W,H    0,0,720,220
    Framerate:    6246/250
    Field:    1

 

The result are:
[My app/simple_5_transcode_opaque_async_vppresize]

simple_5_transcode_opaque_async_vppresize.png

[sample_multi_transcode]

sample_multi_transcode.png

Both apps are running on the same system, same input file. They both use OpaqueSurface.
I tried to trace down the call stacks on both applications to see if there are any difference when they initialized. The call sequences are nearly identical! At this point, I am pretty much scratching my head off. Any help you can help would greatly appreciated!

[System Info]
OS: SLES12
CPU: Intel XEON E3-1285
MSDK: 2015R3

Input file is h264.

Attached is a modified simple_5_transcode_opaque_async_vppresize that includes the identical ResizeAndLetterBox routine used in both apps (one works and one doesn't). Look for doResizeNBoxVPP() routine.

Many thanks in advance,
Ken S.

0 Kudos
4 Replies
Highlighted
37 Views

Hi Ken,

In the sample, the output of VPP params are fed into the input of the encode params (crop values, width and height). In the tutorial code that you have, that is not the case(crop values differ between vppOut and EncIn). If you match them up, you will see the letterboxing behavior.

Let me know if that fixes the issue - I was able to achieve the effect by the above-mentioned modification. If you are not, we can dig deeper.

In the sample: MSDK_MEMCPY_VAR(m_mfxEncParams.mfx.FrameInfo, &m_mfxVppParams.vpp.Out, sizeof(mfxFrameInfo));

This is missing in the tutorial.

0 Kudos
Highlighted
Beginner
37 Views

Hi Sravanthi,

Thanks for your quick response. I tried that before. In fact, I tried a lot of different CropX,Y,W,H settings and none of them worked. Unfortunately that won't fix the problem.

This is what I did,
    memcpy(&mfxEncParams.mfx.FrameInfo, &VPPParams.vpp.Out, sizeof(mfxFrameInfo));
    mfxEncParams.mfx.FrameInfo.CropX = 0; // adjust the Crop X, Y coordinate, otherwise, the video gets shifted
    mfxEncParams.mfx.FrameInfo.CropY = 0;

Now the mfxEncParams will have these settings:
Encoder Param: 
      CodecId: 541283905
      Profile: 66
        Level: 31
  TargetUsage: 4
GopSize(flag): 50(1)
   IpInterval: 1
  IdrInterval: 0
  RateControl: 1
    FrameInfo:
         WxH (AR): 720x224 (0:0)
      CropX,Y,W,H: 0,0,390,220

        Framerate: 6246/250
            Field: 1

The result screenshot is as follow:
sample_multi_transcode_2.png

Although it does preserves the aspect ratio of the video, there are significant problems:
1) It doesn't do letterbox (in this case, it doesn't do pillarbox)
2) if you ffprobe the video, the video size is 390x220, NOT 720x220
    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 390x220 [SAR 1:1 DAR 39:22], 760 kb/s, 24.98 fps, 24.98 tbr, 1200k tbn, 49.97 tbc (default)
3) when I played it back with vlc, it gave me a waring:
[h264 @ 0x7f5c94c7cd20] brainfart cropping not supported, this could look slightly wrong ...

Most importantly as I mentioned before, the SAME logic works CORRECTLY with sample_multi_transcode.  One thing I didn't clarify in my earlier post is that sample_multi_transcode itself DOES NOT do resize and letterbox. I need to add the same logic "doResizeNBoxVPP() " to sample_multi_transcode  as well. And, when I did that, it works as desired. Unfortunately, the SAME logic doesn't work in simple_5_transcode_opaque_async_vppresize.

When you say, you get a correct letterbox behavior, can you kindly share the code you modified with me? Perhaps I am making some obvious mistakes.

Thanks,
Ken S.

 

 

0 Kudos
Highlighted
Beginner
37 Views

Hi Sravanthi,

I finally tracked down the problem. It was the Frame Surface on the tutorial sample code. Basically when we created the VppOut FrameSurface, the VppOut FrameInfo structure should be used instead of the EncIn FrameInfo. In most cases, VppOut == EncIn. But, for my scenario (resize and crop), the FrameInfo is different between VppOut and EncIn.


    mfxFrameSurface1** pSurfaces2 = new mfxFrameSurface1 *[nSurfNumVPPEnc];
    MSDK_CHECK_POINTER(pSurfaces2, MFX_ERR_MEMORY_ALLOC);
    for (int i = 0; i < nSurfNumVPPEnc; i++) {
        pSurfaces2 = new mfxFrameSurface1;
        MSDK_CHECK_POINTER(pSurfaces2, MFX_ERR_MEMORY_ALLOC);
        memset(pSurfaces2, 0, sizeof(mfxFrameSurface1));
        //memcpy(&(pSurfaces2->Info), &(EncRequest.Info), sizeof(mfxFrameInfo));
        memcpy(&(pSurfaces2->Info), &(VPPParams.vpp.Out), sizeof(mfxFrameInfo));
    }

After I changed it to use VppOut FrameInfo, everything works as expected.


Thanks,
Ken S.

0 Kudos
Highlighted
37 Views

Thanks for updating the post Ken! Glad you got it working. When using a media pipeline (dec-vpp-enc), using the correct parameters is probably the tricky part! Thanks again.

0 Kudos