Media (Intel® Video Processing Library, Intel Media SDK)
Access community support with transcoding, decoding, and encoding in applications using media tools like Intel® oneAPI Video Processing Library and Intel® Media SDK
Announcements
The Intel Media SDK project is no longer active. For continued support and access to new features, Intel Media SDK users are encouraged to read the transition guide on upgrading from Intel® Media SDK to Intel® Video Processing Library (VPL), and to move to VPL as soon as possible.
For more information, see the VPL website.

Frame Rate Conversion

babgvant
Beginner
1,018 Views

I've been playing with the sample_multi_transcode sample, trying to extend it so that it actually supports framerate conversion but I haven't been seeing results consistent with what I expect (frame count delivered to decoder = frames count output by encoder) when output framerate is set to 1/2 input framerate. Am I doing this incorrectly or are my expectations incorrect (if so why)?

Thanks

The code I added is below:

// configure and attach external parameters
    if(m_bConvertFrameRate)
    {
        mfxStatus sts = AllocAndInitVppDoUse();
        MSDK_CHECK_RESULT(sts, MFX_ERR_NONE, sts);
        m_VppExtParams.push_back((mfxExtBuffer *)&m_VppDoUse);

        m_FrameRateConversion.Algorithm = MFX_FRCALGM_DISTRIBUTED_TIMESTAMP;
        m_VppExtParams.push_back((mfxExtBuffer *)&m_FrameRateConversion);
    }
...
    
    m_mfxVppParams.ExtParam = &m_VppExtParams[0]; // vector is stored linearly in memory
    m_mfxVppParams.NumExtParam = (mfxU16)m_VppExtParams.size();

...

mfxStatus CTranscodingPipeline::AllocAndInitVppDoUse()
{    
    m_VppDoUse.NumAlg = 1;

    m_VppDoUse.AlgList = new mfxU32 [m_VppDoUse.NumAlg];    
    MSDK_CHECK_POINTER(m_VppDoUse.AlgList,  MFX_ERR_MEMORY_ALLOC);

    m_VppDoUse.AlgList[0] = MFX_EXTBUFF_VPP_FRAME_RATE_CONVERSION; // turn on FRC
    
    return MFX_ERR_NONE;

}

0 Kudos
13 Replies
Petter_L_Intel
Employee
1,018 Views

Hi,

It looks like there may be an issue with using FRC with this algorithm enabled. We are exploring this further to find the root cause.

Are you encountering repeated MFX_ERR_MORE_DATA return code when calling RunFrameVPPAsync()?

As a side note. You can still do FRC but without timestamp interpolation. Just skip pushing m_VppDoUse and m_FrameRateConversion extended buffers to the VPP extended buffer list. However, observe the behavior of RunFrameVPPAsync() for the case when converting frame rate from from instance 30 to 60 fps. This will effectively duplicate each frame, and for each duplicated frame, RunFrameVPPAsync() will require additional output surface, indicated by the return code MFX_ERR_MORE_SURFACE.

Regards,
Petter 

0 Kudos
babgvant
Beginner
1,018 Views

Thanks. I'm actually seeing the same behavior (frames in = frames out) no matter which algorithm is selected. I haven't tried increasing the framerate, just stepping down from 60->30 and AFAIK there's no explicit error being generated.

Are you saying that I should handle the FRC myself (dropping/repeating frames) in RunFrameVPPAsync()?

0 Kudos
Petter_L_Intel
Employee
1,018 Views

Hi again,

So it turns out using FRC with the MFX_FRCALGM_DISTRIBUTED_TIMESTAMP option is working as expected after all.

There is however an undocumented requirement to ensure correct behavior (we will work to improve the description in future release of Media SDK). User must provide TimeStamp (PTS) that are increasing to ensure accurate behavior, if not there may be unwanted sideeffects like the one we both observed where repeated MFX_ERR_MORE_DATA status is returned.

Here is an example for FRC from 10fps to 29.97fps:
inPTS    = 0, 9000, 18000, 27000, … (corresponds 10 fps)
outPTS  = 0, 3003, 6006, 9009, … (corresponds 29.97 fps).

Also make sure you handle MFX_ERR_MORE_SURFACE for the case of increasing frame rate as described in earlier post.

Let us know if you are still encountering issues.

Regards,
Petter 

0 Kudos
babgvant
Beginner
1,018 Views

dup

0 Kudos
babgvant
Beginner
1,018 Views

I'm actually trying to go the other way (reduce 60FPS to 30FPS). The use of MFX_FRCALGM_DISTRIBUTED_TIMESTAMP may have been misleading, when I couldn't get it working with MFX_FRCALGM_PRESERVE_TIMESTAMP I started trying the other options.

My code is setting the PTS (based on what I'm getting from the container), but I'm not seeing any change on the other side. I haven't tried generating my own PTS though.

When converting from 60->30 I expect to see a 50% drop in the number of frames received from the encoder, but that's not happening. If I send in 5000 I get 5000 out.

0 Kudos
babgvant
Beginner
1,018 Views

Added some code to generate PTS, and now I'm seeing the 50% reduction in frames.

Thanks

0 Kudos
babgvant
Beginner
1,018 Views

Scratch that. I forgot to comment out my [rough] FRC code.

0 Kudos
Petter_L_Intel
Employee
1,018 Views

Hi,

are you still encountering issues?

When reducing frame rate, note that RunVPPFrameAsync will return MFX_ERR_MORE_DATA indicating that frame(s) are skipped and not passed on to encoder (for instance using a transcode pipeline).

Regards,
Petter 

0 Kudos
babgvant
Beginner
1,018 Views

Yes, I'm not seeing that behavior (no frames are dropped unless I drop them explicitly).

0 Kudos
Petter_L_Intel
Employee
1,018 Views

Hi,

For the case of just dropping frames such as in the case 60 -> 30 fps, you do not have to apply any special settings or extended buffers to VPP. If you look at the Media SDK 2013 sample_multi_transcode code and use a command line such as below, the only thing you have to do to enable 60 -> 30 FRC is the following:

"sample_multi_transcode.exe -i::h264 in.264 -o::h264 out.264 -b 500 -f 60 -hw"

1)
In "VPPPreInit" comment out the 3 lines of code that only enable VPP in the case of resolution change or de-interlace

   //if ( (m_mfxDecParams.mfx.FrameInfo.CropW != pParams->nDstWidth && pParams->nDstWidth) ||
   // (m_mfxDecParams.mfx.FrameInfo.CropH != pParams->nDstHeight && pParams->nDstHeight) ||
   // (pParams->bEnableDeinterlacing))

2) In "InitVppMfxParams" make sure to set output frame rate to 30 such as:
   m_mfxVppParams.vpp.Out.FrameRateExtN /= 2;
 

Regards,
Petter 

0 Kudos
babgvant
Beginner
1,018 Views

Thanks Petter.

I had already modified VPPPreInit to not disable VPP in those cases (by detecting if FRC is necessary earlier). It appears that this code block (from InitDecMfxParams):

// if frame rate specified by user set it for decoder and the whole pipeline
if (pInParams->dFrameRate)
{
     ConvertFrameRate(pInParams->dFrameRate, &m_mfxDecParams.mfx.FrameInfo.FrameRateExtN, &m_mfxDecParams.mfx.FrameInfo.FrameRateExtD);
 } 

was making it so VPP didn't think it needed to drop frames (I had borrowed the -f argument). After commenting it out I am now seeing the behavior I expected.

Thanks again for the assistance.

0 Kudos
Noam_K_
Beginner
1,018 Views

Hi,

I'm using the one of the transcode tutorials and trying to change the video frame rate from 60 to 30 fps.

at some point, before all the input frames are decoded, I'm getting repeated MFX_ERR_MORE_DATA status. is it still a known problem? how can I convert the frame rate successfully? I'm ok with a simple algorithm that is dropping every 2nd input frame...

Thanks,

Noam

 

 

 

0 Kudos
Surbhi_M_Intel
Employee
1,018 Views

Hi Noam,

You can refer to the frame rate conversion article - https://software.intel.com/en-us/articles/frame-rate-conversion-frc-in-intel-media-sdk to decrease the frame rate from 60 to 30fps. Here it is explained with the help of the tutorials.
You can check more technical articles over here.

Thanks,
-Surbhi

0 Kudos
Reply