Can I use FRC with MFX_FRCALGM_FRAME_INTERPOLATION on Linux?

[Minwoo_J_]

Hi,

I'm looking for frame rate conversion library including frame interpolation that work on linux. So I applied the Intel Media Server Studio Essentials Edition(2015R2) to my system, and set the MFX_FRCALGM_FRAME_INTERPOLATION flag to frame rate conversion filter. However, I just got the MFX_WRN_FILTER_SKIPPED as return status from the MFXVideoVPP_Init() function.

My system is QuickFire Networks T-Video Series, and OS is Ubuntu 12.04. Is not the MFX_FRCALGM_FRAME_INTERPOLATION flag supported on my system?

[Sravanthi_K_Intel]

Hello there - Thanks for the question. Couple of things:

- Can you provide more details on the code and how you implemented the FRC? It is highly recommended you use our existing samples/tutorials (specifically simple_vpp tutorial which work on both Windows and Linux) and modify them to add FRC and paste the snippet. That will greatly help us.

- We have an article written for FRC here - https://software.intel.com/en-us/articles/frame-rate-conversion-frc-in-intel-media-sdk. The article uses simple_vpp to demonstrate FRC algorithm. Can you please use the code example in this article, and check if you see similar status being returned?

Let me know if you have more questions.

[Minwoo_J_]

Thank you for the comment.

For the test, I simply modified sample_vpp.

In sample_vpp_utils.h, I added FRC config to the sAppResources structure.

struct sAppResources
{
  CRawVideoReader*    pSrcFileReaders[MAX_INPUT_STREAMS];
  mfxU16              numSrcFiles;
  CRawVideoWriter*    pDstFileWriter;

  sFrameProcessor*    pProcessor;
  mfxVideoParam*      pVppParams;
  sMemoryAllocator*   pAllocator;

  /* VPP extension */
  mfxExtVPPDoUse      extDoUse;
  mfxU32              tabDoUseAlg[ENH_FILTERS_COUNT];
  mfxExtBuffer*       pExtBuf[1+ENH_FILTERS_COUNT];
  mfxExtVppAuxData    extVPPAuxData;

  /* config video enhancement algorithms */
  mfxExtVPPProcAmp    procampConfig;
  mfxExtVPPDetail     detailConfig;
  mfxExtVPPDenoise    denoiseConfig;
  mfxExtVPPImageStab  istabConfig;
  mfxExtVPPComposite  compositeConfig;
/*****************Inserted******************************************/
mfxExtVPPFrameRateConversion  frconvConfig;
/*******************************************************************/
};

Then, I inserted the configuration of FRC filter to sample_vpp_utils.cpp in order to unconditionally use the FRC filter like following.

mfxStatus ConfigVideoEnhancementFilters( sInputParams* pParams, sAppResources* pResources )
{
    // (skip)
    
    // [1] video enhancement algorithms can be enabled with default parameters
/*****************Inserted******************************************/
    pResources->tabDoUseAlg[enabledFilterCount++] = MFX_EXTBUFF_VPP_FRAME_RATE_CONVERSION;
/*******************************************************************/

    // (skip)
    
    // [2] video enhancement algorithms can be configured
/*****************Inserted******************************************/
    pResources->frconvConfig.Header.BufferId = MFX_EXTBUFF_VPP_FRAME_RATE_CONVERSION;
    pResources->frconvConfig.Header.BufferSz = sizeof(mfxExtVPPFrameRateConversion);
    pResources->frconvConfig.Algorithm = MFX_FRCALGM_FRAME_INTERPOLATION;
    pVppParam->ExtParam[pVppParam->NumExtParam++] = (mfxExtBuffer*)&(pResources->frconvConfig);
/*******************************************************************/

    // (skip)
}

Finally, I added check code to InitFrameProcessor() function in sample_vpp_utils.cpp.

mfxStatus InitFrameProcessor(sFrameProcessor* pProcessor, mfxVideoParam* pParams)
{
  mfxStatus sts = MFX_ERR_NONE;

  MSDK_CHECK_POINTER(pProcessor,          MFX_ERR_NULL_PTR);
  MSDK_CHECK_POINTER(pParams,             MFX_ERR_NULL_PTR);
  MSDK_CHECK_POINTER(pProcessor->pmfxVPP, MFX_ERR_NULL_PTR);

  // close VPP in case it was initialized
  sts = pProcessor->pmfxVPP->Close();
  MSDK_IGNORE_MFX_STS(sts, MFX_ERR_NOT_INITIALIZED);
  MSDK_CHECK_RESULT(sts,   MFX_ERR_NONE, sts);

  // init VPP
  sts = pProcessor->pmfxVPP->Init(pParams);
/*****************Inserted******************************************/
  printf("Check mfxVPP Init Return = %d\n", sts);
/*******************************************************************/
  MSDK_IGNORE_MFX_STS(sts, MFX_WRN_PARTIAL_ACCELERATION);
  MSDK_CHECK_RESULT(sts,   MFX_ERR_NONE, sts);

  return MFX_ERR_NONE;

} // mfxStatus InitFrameProcessor(sFrameProcessor* pProcessor, mfxVideoParam* pParams)

The following is the test log.

root@V2200-Demo:/opt/intel/mediasdk/samples/mybin/__bin# ./sample_vpp_drm -lib hw -sw 1920 -sh 1080 -sf 30.0 -scc nv12 -spic 1 -dw 1920 -dh 1080 -df 60.0 -dcc nv12 -dpic 1 -n 20 -i NV12.yuv -o out.yuv
libva info: VA-API version 0.34.0
libva info: va_getDriverName() returns 0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/iHD_drv_video.so
libva info: Found init function __vaDriverInit_0_32
libva info: va_openDriver() returns 0
Check mfxVPP Init Return = 10
Intel(R) Media SDK VPP Sample Version 0.0.000.0000

Input format	NV12
Resolution	1920x1088
Crop X,Y,W,H	0,0,1920,1080
Frame rate	30.00
PicStruct	progressive
Output format	NV12
Resolution	1920x1088
Crop X,Y,W,H	0,0,1920,1080
Frame rate	60.00
PicStruct	progressive

Video Enhancement Algorithms
Denoise		OFF
VideoAnalysis	OFF
ProcAmp		OFF
Detail		OFF
ImgStab		OFF

Memory type	system

MediaSDK impl	hw
MediaSDK ver	1.10

VPP started
Frame number: 40
VPP finished

Is there any problem on my test?

[Sravanthi_K_Intel]

Hello there-

Based on your command line, you are specifying the # frames as 20 (-n 20), and the output log shows that the frames output were 40. Can you please confirm you are not getting 40 frames in the output file (out.yuv)?

My experiment on Cent7.0 (with MSS R3) shows filter skipped as well, BUT with the output frames and input frames = 20. I will investigate further, but can you please confirm if out.yuv =40 frames (FRC interpolated) and NV12.yuv = 20?

[Minwoo_J_]

I appreciate your reply.

I got 40 frames on my test and the output file had duplicated frames.

Today, I tried more experiments for this problem.

First, I applied same code to windows in order to confirm whether the code is correct. I got the same result like on linux. Therefore I modified simple_vpp.cpp as you say on first comment for getting more certain code. However, I  unfortunately received the same result with the first test on both windows and linux. How can I  use FRC with interpolation? Please help me.

[Sravanthi_K_Intel]

Hello there,

I tested the algorithm on Windows, and I observe similar behavior - duplication instead of interpolation. And on Linux, I see the filter being skipped. I have reported this behavior to our team, and as soon as we have more information on this, I will update you. Thanks for bringing this to our attention.

[Adrian_R_1]

I am seeing similar behaviour on Linux with SDK version 1.21

I followed the guide on https://software.intel.com/en-us/articles/frame-rate-conversion-frc-in-intel-media-sdk to double the frame rate of a sample RAW file (30-60). Other than what is in the guide I explicitly set the API version 1.21 and also declared
bool isMultipleOut = false;
which seemed to be missing.

 If (as in the guide) I run with MFX_FRCALGM_DISTRIBUTED_TIMESTAMP
m_FrameRateConversion.Algorithm = 2;
I get duplicated frames with correct timestamps.  Similarly, with Algorithm 1 (MFX_FRCALGM_PRESERVE_TIMESTAMP) you get duplicate frames with every other frame's timestamp set to -1, as described in the docs.

Running with just MFX_FRCALGM_FRAME_INTERPOLATION I get what seems to be identical behaviour to MFX_FRCALGM_PRESERVE_TIMESTAMP (i.e. frames are simply duplicated and every other frame has timestamp -1).

From the docs on FrcAlgm, I would expect to be able to run with Algorithm = 6 (i.e. MFX_FRCALGM_DISTRIBUTED_TIMESTAMP | MFX_FRCALGM_FRAME_INTERPOLATION) but if I do so I get an MFX_ERR_INVALID_VIDEO_PARAM error.  Given the API version I am using, I would at the least expect it to return MFX_WRN_INCOMPATIBLE_VIDEO_PARAM instead...

I would be more than happy to attach my patched version of simple_vpp.cpp if that would help.