Enabling EncTools h264 in oneVPL

[Max_Gud]

Hello, there was a question about enabling EncTools for the H264 codec in oneVPL, I found the mfxenctools.h file in the oneVPL-gpu repository, attached it to the project and connect the external buffer mfxExtEncToolsConfig to mfxVideoParam, turn on the parameters and it works, gives a gorgeous picture with Lookahead enabled in CBR bitrate control mode with ExtBRC, but there is a problem - after some time, when doing video encoding, I get frame drops, the time before the start of frame drops varies from 40 minutes to 1 hour 40 minutes. Frame drops start regardless of the settings set, it is enough to attach mfxExtEncToolsConfig to mfxVideoParams and they appear. I want to ask if I am connecting incorrectly or if there are any restrictions on using EncTools. I ask a question, because, unfortunately, there is no documentation on EncTools at all.

Hardware Used:
CPU: 12700K with UHD770
Intel ARC A380.

Полный код приложения доступен по ссылке: obs-qsv-onevpl/obs-qsv-onevpl-encoder-internal.cpp at dev · ma3uk/obs-qsv-onevpl · GitHub

The connection code is shown below:

INIT_MFX_EXT_BUFFER(mfx_EncToolsConf,

    MFX_EXTBUFF_ENCTOOLS_CONFIG);

mfx_EncToolsConf.AdaptiveI = MFX_CODINGOPTION_ON;

mfx_EncToolsConf.AdaptiveB = MFX_CODINGOPTION_ON;

mfx_EncToolsConf.SceneChange = MFX_CODINGOPTION_ON;

mfx_EncToolsConf.AdaptivePyramidQuantP = MFX_CODINGOPTION_ON;

mfx_EncToolsConf.AdaptivePyramidQuantB = MFX_CODINGOPTION_ON;

mfx_EncToolsConf.AdaptiveQuantMatrices = MFX_CODINGOPTION_ON;

mfx_EncToolsConf.AdaptiveMBQP = MFX_CODINGOPTION_ON;

mfx_EncToolsConf.AdaptiveLTR = MFX_CODINGOPTION_ON;

mfx_EncToolsConf.AdaptiveRefP = MFX_CODINGOPTION_ON;

mfx_EncToolsConf.AdaptiveRefB = MFX_CODINGOPTION_ON;

mfx_EncToolsConf.BRCBufferHints = MFX_CODINGOPTION_ON;

mfx_EncToolsConf.BRC = MFX_CODINGOPTION_ON;

mfx_ENC_ExtendedBuffers.push_back(

(mfxExtBuffer *)&mfx_EncToolsConf);

[Max_Gud]

You may not be able to reproduce the error due to what you are trying to do on a Linux system.
In turn, I was able to reproduce it on 3 Windows-based systems with different hardware configurations, but all configurations were united by one thing, that they worked with Intel Arc A380 and A750 video cards. I hope you will still listen to me and try to reproduce it on a Windows system.

[Max_Gud]

Also, after additional tests, I want to say that this problem also affects integrated GPUs. Frame drops also occur on my UHD 770, but later than on the Arc A380

[Max_Gud]

I was able to run the checks on a Linux-based system. While conducting tests, I came across the fact that when I try to use EXTBRC + Lookahead, I get the error MFX_ERR_INVALID_VIDEO_PARAM on AV1 and AVC codecs, the only codec on which EXTBRC + Lookahead sharing works - this is HEVC, which correlates with the results obtained in tests on a system running Windows, with the only difference that on a Windows system, the combined use of these parameters does not cause an MFX_ERR_INVALID_VIDEO_PARAM error on AVC and AV1 codecs, but causes frame drops after a certain time and the HEVC codec is the only one that is not subject to frame drops.

I will remind you that I do not use FFMPEG, I use pure "vanilla" oneVPL with a new memory allocator

[JaideepK_Intel]

Hi,

Sorry for the delay. We were able to run the command on Windows via webcam. I am attaching the command below for your reference.

ffmpeg -f vfwcap -use_wallclock_as_timestamps 0 -rtbufsize 500M -thread_queue_size 8192 -framerate 60 -i video="HP HD Camera" -c:v h264_qsv -async_depth 2 -bitrate_limit 0 -low_power true -look_ahead_depth 40 -extbrc 1 -b_strategy 1 -adaptive_i 0 -adaptive_b 0 -bf 7 -strict 1 -r 60 -rdo 1 -mbbrc 1 -b_strategy 1 -idr_interval 0 -look_ahead 1 -look_ahead_downsampling off -bufsize 16M -maxrate 8M -minrate 8M -rc_init_occupancy 8M -profile:v high -b:v 8M -c:a copy "test.mp4"

Could you please share the following details?

At what time are you observing frame drops, and could you please share an approximate time?

Have you tried running the same command on other machines (except A380s, A750s, and UHD 770) and observed the same issue? If so, please share those hardware details.

Thanks,

Jaideep

[Max_Gud]

A frame drop can occur at random time during the recording or broadcast period from 15 minutes to 60 minutes with an increase in CPU load - starting a game, launching a Youtube video in the browser, launching a video in PotPlayer, in general, any task that creates a load on the processor. Also, a drop of frames occurs when idle, i.e. if you turn on recording and move away from the computer for a few hours, then drops will still begin after 30-120 minutes, or earlier, perhaps in this case, some background task creates a load on the CPU and this is a trigger.
In general, there is no specific time after which drops begin, but in the end they always appear.
I have not tried to run on other equipment other than the above.

[JaideepK_Intel]

Hi,

Sorry for the delay, We need some info like what settings you are using while streaming via OBS. We are attaching a screenshot from our end.

Please let us know if we are missing anything.

In video encoder settings, we gave the below flags.

-extra_hw_frames 40 -preset 1 -async_depth 2 -bitrate_limit 0 -low_power true -look_ahead_depth 40 -extbrc 1 -b_strategy 1 -adaptive_i 0 -adaptive_b 0 -bf 7 -strict 1 -r 60 -rdo 1 -mbbrc 1 -b_strategy 1 -idr_interval 0 -look_ahead 1 -look_ahead_downsampling off -profile high

We also ran the FFMPEG command (encoding with qsv on arc 380), which was shared in our previous post, for around 6 minutes, but haven't seen any frame drops. attaching screenshot for your reference. During this process, we ran some heavy tasks on the GPU and CPU.

It would be really helpful for us if, you share the exact steps that we need to follow on OBS (a video explaining the exact steps is much better)

Thanks,

Jaideep

[Max_Gud]

Your screenshot of using ffmpeg shows that in 17 seconds of recording you received 554 duplicate frames, as well as a memory leak of more than 6 gigabytes and you say that you did not detect the problem.

Also in the OBS Recording section, you have selected rawvideo codec, which is not the h264_qsv codec, it does not use video card resources and runs on the CPU.

I will try to describe to you in detail the steps to reproduce the problem in the next post, please follow the steps clearly so that there are no problems with playback.

[Max_Gud]

1. Download the archive "obs-qsvonevpl_dev6.zip" with the encoder plugin for OBS with ExtBRC support from this link - Release 1.1.0.0_dev5 · ma3uk/obs-qsv-onevpl (github.com)
2. Unzip the contents of the downloaded archive with the encoder to the folder at the location of OBS Studio \obs-plugins\64 bit
3. Launch OBS Studio
4. Go to Settings
5. Go to the "Output" settings section
6. Set the "Output mode" parameter to the "Advanced" state
7. Go to the "Recording" tab
8. Select the "Video encoder" option in the QuickSync oneVPL H.264 state
9. Set the "TargetUsage" parameter to the "TU1 (Veryslow)" state
10. Set the "ExtBRC" parameter to the "IMPLICIT" state
11. Set the "Bitrate" parameter to 8000
12. Set the "Lookahead" parameter to the "ON" state
13. Set the "LookaheadDownSampling" parameter to the "Slow" state
14. Set the "Lookahead latency" parameter to the "HIGH"
state

15. Additionally, you can set the slider of the NumRefFrame parameter to the 15 position
16. Save the settings by pressing the "Apply" key.
17. Go to the "Video" menu on the left side of the OBS Studio settings window
18. Set the Base (Canvas) Resolution and Output (Canvas) Resolution parameters to 1920x1080.
20. Set the FPS parameter to the "Common FPS Value" state and specify the value of 60 FPS.
21. Confirm saving the settings by clicking the "Apply"
button

22. Close the settings window
23. In the "Source" window, add the "Media Source" source with any source name
24. Confirm the creation of the source by clicking "OK"
25. In the "Local File" field, select the path to the video recording of the sample that I sent earlier
26. Check the "Loop" checkbox
27. Check the box "Use hardware decoding when available"
28. Confirm the creation of the source by pressing the "OK" key
29. Make the created source with the sample video visible
30. Adjust the video size to the canvas size of 1920x1080
31. Start recording
32. Press the "Docks" key in the top menu of OBS Studio
33. Check the box for the "Stats" parameter to display recording statistics with the number of encoded frames, as well as the number of missed frames.
34. Leave the recording for 120 minutes and drink tea with cookies and wait until the first digit in the "Skipped fames due to encoding lag" line on the "Stats" panel starts to increase, signaling missed frames.

[JaideepK_Intel]

Hi,

We were able to reproduce your issue from our end, and we are checking on different hardware.

Thank you for your patience.

Regards,

Jaideep

[Max_Gud]

The 4885 driver made some changes, the ExtBRC parameter was broken, now it does not activate EncTools and does not work with LA, causing error -15. Now EncTools PreEncSCD does not work, which is why the quality of the output image has become noticeably worse.
I managed to get PreEncSCD to work with Scenario REMOTE_GAMING and LADepth = 8 and GOPRefDist = 8 using the external buffer mfxExtEncToolsConfig, which was mentioned here earlier and was not recommended by Intel employees for use, with such parameters the quality improves, but the problem with frame drops is not fixed, now they begin to occur later. I managed to reproduce the beginning of the incessant frame loss after 3 hours of continuous recording, but periodic drops occurred starting at the 2 o'clock mark, the trigger for periodic drops was daily work at the PC - switching tabs in the browser, playing videos on Youtube, playing local videos from a PC in AV1 format with processing on the processor, in general, everything is the same the conditions that I described in earlier posts.

[JaideepK_Intel]

Hi,

I hope you are doing well.

Sorry for the delay, Since the issue involves both Intel and non-Intel software, we are having to approach the problem with respect to many variables. We are currently analyzing whether the issue is caused by some interaction between OBS and VPL.

Thanks,

Jaideep

[Max_Gud]

Update after the release of driver 4952. The ExtBRC parameter has not been fixed. The joint work of Lookahead and AdaptiveI was broken. When setting CO3.Scenario = MFX_SCENARIO_GAME_STREAMING, setting LookAheadDepth to 20, 40, 60 or 100, turning on CO2.AdaptiveI frame drops appear within 2-3 minutes of recording, drops appear with H264 and AV1 codecs. There are no drops with the HEVC codec. The problem is only on the Arc A380 video card, there are no drops on the integrated UHD 770 video card

[JaideepK_Intel]

Hi,

I hope you are doing well.

Thanks for sharing the information, we will look into this and get back to you with an update.

Regards,

Jaideep

[JaideepK_Intel]

Hi,

I hope you are doing well and sorry for the delay.

We rootcaused this issue with to bug in EncTools bitrate control that causes gradual growth of processing time, which in turn sooner or later causes frame skip in OBS. Note, that there are two different implementations of EncTools bitrate control – one is affected by this issue another is not. This is affected one

We fixed the issue, verified the fix, and will let you know when the fix will be publicly available. 

Thanks,

Jaideep