FRC algorithm, all src video fps 23.976 -> dst video fps 30 ==> audio not in sync with video

[parksh]

FRC algorithm, all src video fps 23.976 -> dst video fps 30 ==> audio not in sync with video

Hello.

E3-1286 vs i7-11700/i7-12700 FRC vpp hardware transcode issue

all video vps 23.976 -> audio not in sync with video

MSDK 2014 R2
Windows Server 2012 R2
E3-1286

-> SW/HW outputFrameNum ===> 117667 frames 
        ==> 117667 frames

MSDK / oneVPL
Windows 10 Pro for Workstations
i7-11700 / i7-12700
-> SW : E3-1286 same result outputFrameNum ===> 117667 frames
    HW : outputFrameNum ===> 117650 frames

    ===> audio not in sync with video!

i7-11700 MFX_FRCALGM_FRAME_INTERPOLATION flag -> mfxExtVPPFrameRateConversion structure, attach it to the structure, and call the MFXVideoVPP_Query function.
function returns a MFX_ERR_NONE status

===> i7-11700 sample_multi_transcode.exe proc result
sample_multi_transcode.exe -i::h264 input.h264 -o::h264 "output.h264" -FRC::INTERP -f 30.00 -hw
Multi Transcoding Sample Version 8.4.27.0

CONFIGURE LOADER: required implementation: hw
CONFIGURE LOADER: required implementation mfxAccelerationMode: MFX_ACCEL_MODE_VIA_D3D11
CONFIGURE LOADER: Use dispatcher's low-latency mode
Session 0:
Pipeline surfaces number (EncPool): 5
Pipeline surfaces number (DecPool): 8
Loaded Library configuration:
Version: 2.3
ImplName: mfx-gen
Adapter number : 0
Adapter type: integrated
Loaded module path: C:\Windows\System32\DriverStore\FileRepository\iigd_dch.inf_amd64_e648bb2a2af8e9de\libmfx64-gen.dll

Input video: AVC
Output video: AVC

Session 0 was NOT joined with other sessions

Transcoding started
........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
Transcoding finished

Common transcoding time is 1718.62 sec
-------------------------------------------------------------------------------
*** session 0 [00000133BF588660] PASSED (MFX_ERR_NONE) 1718.62 sec, 117650 frames, 68.456 fps
-i::h264 input.h264 -o::h264 output.h264 -FRC::INTERP -f 30.00 -hw

-------------------------------------------------------------------------------

===> Even if I add the filter below, the result is the same
// Implementation must provide VPP scaling
cfg[1] = MFXCreateConfig(loader);
VERIFY(NULL != cfg[1], "MFXCreateConfig failed")
cfgVal[1].Type = MFX_VARIANT_TYPE_U32;
cfgVal[1].Data.U32 = MFX_EXTBUFF_VPP_SCALING;
sts = MFXSetConfigFilterProperty(
cfg[1],
(mfxU8 *)"mfxImplDescription.mfxVPPDescription.filter.FilterFourCC",
cfgVal[1]);
VERIFY(MFX_ERR_NONE == sts, "MFXSetConfigFilterProperty failed");

If tested on Linux OS, is FRC processing normal?

Or is there something I'm missing?

Thanks in advance for your reply.

ps. all 23.976 fps video -> audio not in sync with video

      all ather fps (ex, 24, 29.987, 30, 60..) no problem -> audio / video sync ok

                                                                                                 -> E3-1286 / i7-11700 outputFrameNum same(SW/HW)

[RajashekarK_Intel]

Hi, Thanks for posting in Intel Communites.

Inorder to debug your issue further, can you please provide the following information.

1. oneVPL version

2. oneVPL GPU Runtime version

3. Media driver version:

4. Sample reproducer code which you mentioned it as sample_multi_transcode along with the input video where you are facing this issue.

Regards,

Rajashekar

[parksh]

This is the requested information.

CPU : E3-1286
OS : Windows Server 2012 R2
MSDK : MSDK 2014 R2
API version : 1.10
libmfxhw64.dll
- File Version : 5.14.5.15
- Product Version : 5.0.15333362.93094

CPU : i7-11700
OS : Windows 10 Pro for Workstations
API version : 2.3
oneVPL version : master version(2023.2.1)
libmfx64-gen.dll
- File Version : 11.21.6.24
- Product Version : 11.0.0.004

sample_multi_transcode sample code(build master version run)
-> https://github.com/oneapi-src/oneVPL/tree/master/tools/legacy/sample_multi_transcode

sample video download link

http://naver.me/x6UVtygP 

password : test1234

Thank you!

[RajashekarK_Intel]

Hi,

We could observe output frames as 117650 in both latest version of MediaSDK and oneVPL. Hence we request you to upgrade to the latest version.

With regards to Audio not sync with video, we've addressed this issue in one of your previous thread that "VPL does not do any Audio coding, and AV Sync is usually handled by the container, which is beyond the scope of VPL".

FYI, The below are the outputs of sample_multi_transcode with the input video you've sent.

Let us know if you require any additional information after upgrading to the latest versions.

If this resolves your issue, make sure to accept this as a solution. This would help others with similar issue.

Regards,

Rajashekar

[parksh]

Thanks for the anser

MSDK or oneVPL is Can't the software process using the FRC algorithm on the sample?

In our program MSDK mfxIMPL impl = MFX_IMPL_SOFTWARE;
-> When running in SW mode, the number of frames is 117667

However, when our program is run in MSDK or oneVPL mfxIMPL impl = MFX_IMPL_HARDWARE_ANY | MFX_IMPL_VIA_D3D11;
-> When running in HW mode, the number of frames is 117650

Only 23.976 fps video has this frame gap

Thanks.

[RajashekarK_Intel]

Hi,

In order to understand your issue and debug this, can you provide below information.

>> When running in SW mode, the number of frames is 117667.

1. Can you provide the exact steps and sample reproducer if you made any changes to the existing MSDK sample_multi_transcode sample.

2. We could observe the frames in your screenshot as 117667, but we are unable to replicate it with -sw, can you provide the command, along with the whole log of transcoded the video.

3. Did you upgrade the MSDK version to the latest or you're using the mentioned MSDK 2014 R2 for -sw(software)?

Regards,

Rajashekar

[parksh]

- MSDK sample_multi_transcode sample.

http://naver.me/ID14ayPk

- -sw sample code(MSDK)

http://naver.me/GisMKI7b

-> -sw : TransocdeLauncher.cpp 76 line fix

               mfxIMPL impl = MFX_IMPL_SOFTWARE;

               ==> FRAMECOUNT : 117667

     -hw : TransocdeLauncher.cpp 76 line fix

               mfxIMPL impl = MFX_IMPL_HARDWARE_ANY | MFX_IMPL_VIA_D3D11;

               ==> FRAMECOUNT : 117650

※ DEBUG mode run

- input.h264 video sample

http://naver.me/x6UVtygP

run MSDK version to the latest for -sw(software)

※ link passwd : test1234

Thanks

[RajashekarK_Intel]

Hi,

The "sample_multi_transcode" you shared with us had dependencies which we tried linking manually, but it is observed that there are changes to existing sample (when compared to the one present in oneVPL/tools/legacy/ repository). Hence it is failing to build due to internal dependencies.

Kindly share the below information.

1) Provide the steps to reproduce the sample_multi_transcode you've sent to us (Screenshots on how you are linking from scratch is much appreciated), along with the log where you were able to reproduce -sw acceleration and found 117667 frames for the given video.

2) Visual Studio version along with Windows SDK version.

Thanks,

Rajashekar

[parksh]

I changed the existing sample, but the oneVPL build version is the result of running sample_multi_transcode.exe
Doesn't seem to have anything to do with the issue now

- -sw sample code(MSDK)
-> Windows SDK version : 10.0
VIsual Studio 2019
Intel Media Server Studio 2014 R2 Installed

sample_multi_transcode -sw execution result is good to compare, but an error occurs when the -sw option is executed.

how to sample_multi_transcode run -sw(FRC algorithm)?

Currently, I do not have a Linux environment, but if you have a Linux environment on a CPU of 10th generation or higher,
Can you provide me with the result of running below command in -sw, -hw respectively?
-> sample_multi_transcode.exe -i::h264 input.h264 -o::h264 "output.h264" -FRC::INTERP -f 30.00 -hw

I'm sorry that my English isn't smooth as I use translation.
Thanks

[RajashekarK_Intel]

Hi, Thank you for your detailed response.

We could reproduce that sample_multi_transcode is causing issue when "-sw with FRC::INTERP" enabled on both Linux and windows. while we're trying out the workaround you asked for "-hw with FRC::INTERP". we've informed it to the dev team and we'll get back to you soon with an update.

FYI, with respect to "-hw with FRC::INTERP" option we've already provided you the log of running on windows.

Regards,

Rajashekar

[RajashekarK_Intel]

Hi,

Good day to you.

We've an intermediate update from the dev team that, The software option(-sw) with MediaSDK and oneVPL are both EOLd for -FRC::INTERP and are not expected to work on the recent oneVPL releases on both Linux and Windows.

Regarding -FRC::INTERP -hw on Linux, we'll get back to you once we've an update with the dev team.

Regards,

Rajashekar

[parksh]

Thank you for answer.
I look forward to your update.

[RajashekarK_Intel]

Hi, Good day to you.

Apologies for the delayed response.

Please find the attachment below for sample_multi_transcode run with -FRC::INTERP -hw on linux.

Kindly confirm if this is the information is what you are looking for, and do let us know if we can stop monitoring this thread.

Regards,

Rajashekar

[RajashekarK_Intel]

Hi, Good day to you.

We have not heard back from you, were you able to review the information provided on our previous reply?

Kindly confirm and do let us know if we can stop monitoring this thread.

Regards,

Rajashekar

[parksh]

Hi, Good day to you.

Apologies for the delayed response.

You may close the case.

Thanks again for your help.

[RajashekarK_Intel]

Hi, Thanks for confirming.

If you need any additional information, please post a new question as this thread will no longer be monitored by Intel.

Regards,

Rajashekar