Enabling EncTools h264 in oneVPL

[Max_Gud]

Hello, there was a question about enabling EncTools for the H264 codec in oneVPL, I found the mfxenctools.h file in the oneVPL-gpu repository, attached it to the project and connect the external buffer mfxExtEncToolsConfig to mfxVideoParam, turn on the parameters and it works, gives a gorgeous picture with Lookahead enabled in CBR bitrate control mode with ExtBRC, but there is a problem - after some time, when doing video encoding, I get frame drops, the time before the start of frame drops varies from 40 minutes to 1 hour 40 minutes. Frame drops start regardless of the settings set, it is enough to attach mfxExtEncToolsConfig to mfxVideoParams and they appear. I want to ask if I am connecting incorrectly or if there are any restrictions on using EncTools. I ask a question, because, unfortunately, there is no documentation on EncTools at all.

Hardware Used:
CPU: 12700K with UHD770
Intel ARC A380.

Полный код приложения доступен по ссылке: obs-qsv-onevpl/obs-qsv-onevpl-encoder-internal.cpp at dev · ma3uk/obs-qsv-onevpl · GitHub

The connection code is shown below:

INIT_MFX_EXT_BUFFER(mfx_EncToolsConf,

    MFX_EXTBUFF_ENCTOOLS_CONFIG);

mfx_EncToolsConf.AdaptiveI = MFX_CODINGOPTION_ON;

mfx_EncToolsConf.AdaptiveB = MFX_CODINGOPTION_ON;

mfx_EncToolsConf.SceneChange = MFX_CODINGOPTION_ON;

mfx_EncToolsConf.AdaptivePyramidQuantP = MFX_CODINGOPTION_ON;

mfx_EncToolsConf.AdaptivePyramidQuantB = MFX_CODINGOPTION_ON;

mfx_EncToolsConf.AdaptiveQuantMatrices = MFX_CODINGOPTION_ON;

mfx_EncToolsConf.AdaptiveMBQP = MFX_CODINGOPTION_ON;

mfx_EncToolsConf.AdaptiveLTR = MFX_CODINGOPTION_ON;

mfx_EncToolsConf.AdaptiveRefP = MFX_CODINGOPTION_ON;

mfx_EncToolsConf.AdaptiveRefB = MFX_CODINGOPTION_ON;

mfx_EncToolsConf.BRCBufferHints = MFX_CODINGOPTION_ON;

mfx_EncToolsConf.BRC = MFX_CODINGOPTION_ON;

mfx_ENC_ExtendedBuffers.push_back(

(mfxExtBuffer *)&mfx_EncToolsConf);

[Max_Gud]

I recorded a video in which at 48 minutes and 15 seconds you can see how the frame drops begin, which increase to a maximum at 49 minutes and the video turns into a slideshow. Link to the video. I also want to clarify that such frame drops occur ONLY in MFX_RATECONTROL_LA and in MFX_RATECONTROL_CBR mode with Lookahead enabled via EXTBRC or Scenario LIVE_STREAMING. In other modes, when ExtBufferEncToolsConfig is connected, there are no drops, also, if you include the CPUEncToolsProcessing parameter in MFX_EXT_CONFIGOPTION3 and disable ExtBufferEncToolsConfig, there are no drops either, but the quality of the output video becomes much worse.

[JaideepK_Intel]

Hi,

Thank you for posting in Intel communities.

To understand your query better, please share the details mentioned below.

• The current case is the continuation of the below case?

link: https://community.intel.com/t5/Media-Intel-oneAPI-Video/AV1-with-constant-bitrate-CBR/m-p/1488775#M14671

• Please provide the operating system you are using, and if it is Linux, please mention the distro(ubuntu, RHEL, centos... etc) you are using and also the kernel version.

Thanks,

Jaideep

[Max_Gud]

Hello, no, this is a separate issue unrelated to the previous thread.
Windows 10 22H2 is used

[Max_Gud]

Update! The problem when connecting MFX_EXTBUFF_ENCTOOLS_CONFIG also appears with other codecs, checked with AV1 and HEVC. With the AV1 codec, frame drops appeared after 2 hours, at 2 hours and 24 minutes the recording was interrupted due to a large number of dropped frames. With HEVC, trails began to appear after 1 hour 30 minutes and the recording was interrupted for 1 hour 53 minutes. Drops appear only on modes using Lookahead, without Lookahead all codecs work without problems.

[JaideepK_Intel]

Hi,

Sorry for the delay.

Here is high-level information: https://spec.oneapi.io/versions/latest/elements/oneVPL/source/programming_guide/VPL_prg_encoding.html

EncTools are targeted for Linux Server Use cases and should work on Windows as well but not validated. 

EncTools config internals is not public. The best way to exercise it is by using ExtBRC + Lookahead (LA value > GopRefDist). We recommend ExtBRC with GopRefDist 8 and LA 40. (LA could be any value from 0 to N, internally validated up to 60). We do not recommend connecting EncTools config. 

Please check the best command line page here with ffmpeg samples: https://github.com/intel/media-delivery/blob/master/doc/quality.rst

Note: Can you please modify this obs-qsv-onevpl-encoder-internal.cpp with your code (https://github.com/ma3uk/obs-qsv-onevpl/blob/dev/obs-qsvonevpl/obs-qsv-onevpl-encoder-internal.cpp#L1231) and share it here so we can reproduce the same issue from our end.

Also, please look into MSDK mfx_enctools.h maybe this can also help.

Thanks,

Jaideep

[Max_Gud]

This is very sad, because connecting mfxEncToolsConfig gives incomparably better quality when using LA BRC with all codecs. I really hope that Intel Corporation will develop this set of tools and make a full-fledged API and documentation for EncTools.

Regarding changing the code and posting it here, unfortunately this code cannot work independently because it is part of the plugin for OBS Studio. The only way to reproduce the problem is to put the DLL in the OBS Studio plugins folder and enable any LA BRC and EncTools parameter in the encoder settings.

[JaideepK_Intel]

Hi,

Sorry for the confusion.

1. Enctools though not validated for Windows but should work.
2. Can you invoke enctools using ExtBRC ON + LA 40 method? 
3. The ffmpeg cmd line provided will give the same output as you are "connecting enctools config" method.

Can you share the original clip and frame drops clip?

Thanks,

Jaideep

[Max_Gud]

Hi, yes, it turns on, but the quality is different from what I get when I connect mfxEncToolsConfig. With the mfxEncToolsConfig connection, I get the best picture quality. I can share the source video and the video with frame drops, but the video with frame drops has a lot of weight in view of the fact that frame drops do not start immediately, drops can start both after 30-40 minutes and after 90 minutes, because of this video with frame drops has a size of 3-7 gigabytes.

[JaideepK_Intel]

Hi,

Could you please share those files (via link by uploading to your drive or dropbox) ? so we reproduce the same from our end.

Thanks,

Jaideep

[Max_Gud]

Yes, no problem. Here is a link to the video with frame drops - https://drive.google.com/file/d/1dMOk0RpXa6Tqz53S1uyOgt5DI46VUTa2/view?usp=drive_link . And here is the link of the video that I used to record - https://drive.google.com/file/d/1S2zfcDhY75DeHltdv43VWdHcqszQYr8Q/view?usp=drive_link . The video source is short, so I ran it on repeat until the drops appeared.

[Max_Gud]

deleted

[JaideepK_Intel]

Hi,

I hope you are doing well.

Thank you for sharing the links, is it possible to change access to everyone?

manage access-> Everyone with this link

after changing a new link is going to be generated for both videos please share them.

Thanks,

Jaideep

[Max_Gud]

delete

[Max_Gud]

delete

[Max_Gud]

delete

[Max_Gud]

I opened access.

Video with frame drops - https://drive.google.com/file/d/1dMOk0RpXa6Tqz53S1uyOgt5DI46VUTa2/view?usp=sharing

Video source - https://drive.google.com/file/d/1S2zfcDhY75DeHltdv43VWdHcqszQYr8Q/view?usp=sharing

[mkpatel3]

Hi Max, 

Can you please comment which time stamp onwards issue starts in 54 min long video to save debug time? If there are multiple occurrences of the issue, please highlight those time stamps as well. 

One more thing, to replicate the issue, can you also comment Encode Parameter used/configured? (Bitrate, RateControlMode, LA?, EncTools?, GopSize, GopDistance, IDR Interval, TargetUsage etc) please? More information will be better to replicate it via ffmpeg. 

Regards.

[Max_Gud]

At 47 minutes and 30 seconds, you can see that there are missing frames in the video, by the end of the video there are more and more of them and eventually turns into a slideshow
FrameRateExtN- 60 frame
FrameRateExtD - 1

CodecId - MFX_CODEC_AVC
CodecProfile - MFX_PROFILE_AVC_HIGH
GopPicSize - 180
Bitrate - 8000
RateControl - MFX_RATECONTROL_CBR
CodecLevel - AUTO
NumRefFrame - 15 (A value of 16 causes a driver error)
LA - 40, 60, 80, 100 (the more, the faster the frame drops appear, but the time difference is small)
GopOptFlag - MFX_GOP_CLOSED (also with STRICT)
GopRefDist - 4, 8, 16 (the smaller, the faster they disappear)
IdrInterval - 0
NumSlice - 1
TargetUsage - 1 or quality
FourCC - MFX_FOURCC_NV12 (also with the introduction of P010)
ChromaFormat - MFX_CHROMAFORMAT_YUV420

PicStruct - MFX_PICSTRUCT_PROGRESSIVE
Shift - 0
BitDepthChroma - 8
BitDepthLuma - 8
LowPower - ON
And MFX_EXTBUFF_ENCTOOLS_CONFIG is connected without parameters, which in theory should define all its parameters parameters as MFX_CODINGOPTION_UNKNOWN

[mkpatel3]

Hi Max, 

We evaluated bitstream provided and failing point where Frame Drops in Video are observed. Based on analysis of stream, we believe several Repeat Frames are passed to Encoder, Encoder is encoding repeat frames when issue starts (Gives Frame Drop impression). We recommend you check in your infrastructure and see what is going on when issue happens and why new frames are not fed to Encoder at regular interval. 

We tried to replicate the same with ffmpeg and did not observe any issue. 
ffmpeg -qsv_device /dev/dri/renderD129 -c:v av1_qsv -extra_hw_frames 40 -an -stream_loop 135 -i input_2023-05-22_20-49-02.ivf -c:v h264_qsv -preset 1 -async_depth 2 -b:v 8000000 -maxrate 8000000 -minrate 8000000 -bitrate_limit 0 -bufsize 16000000 -rc_init_occupancy 8000000 -low_power true -look_ahead_depth 40 -extbrc 1 -b_strategy 1 -adaptive_i 0 -adaptive_b 0 -bf 7 -g 180 -strict 1 -r 60 -c:a copy avc_ffmpeg_8mbps_g180r8la40_hrd1_loop135.mp4

[Max_Gud]

Why are you using FFMPEG for verification? I do not use FFMPEG and checking with it will not give repeatability of results. The problem occurs only when using oneVPL with MFX_EXTBUFF_ENCTOOLS_CONFIG connection and only when Lookahead value > 0, which cannot be done with FFMPEG. In addition, you are using a Linux system for testing, although the problem occurs on a Windows system