Enabling EncTools h264 in oneVPL

[Max_Gud]

Hello, there was a question about enabling EncTools for the H264 codec in oneVPL, I found the mfxenctools.h file in the oneVPL-gpu repository, attached it to the project and connect the external buffer mfxExtEncToolsConfig to mfxVideoParam, turn on the parameters and it works, gives a gorgeous picture with Lookahead enabled in CBR bitrate control mode with ExtBRC, but there is a problem - after some time, when doing video encoding, I get frame drops, the time before the start of frame drops varies from 40 minutes to 1 hour 40 minutes. Frame drops start regardless of the settings set, it is enough to attach mfxExtEncToolsConfig to mfxVideoParams and they appear. I want to ask if I am connecting incorrectly or if there are any restrictions on using EncTools. I ask a question, because, unfortunately, there is no documentation on EncTools at all.

Hardware Used:
CPU: 12700K with UHD770
Intel ARC A380.

Полный код приложения доступен по ссылке: obs-qsv-onevpl/obs-qsv-onevpl-encoder-internal.cpp at dev · ma3uk/obs-qsv-onevpl · GitHub

The connection code is shown below:

INIT_MFX_EXT_BUFFER(mfx_EncToolsConf,

    MFX_EXTBUFF_ENCTOOLS_CONFIG);

mfx_EncToolsConf.AdaptiveI = MFX_CODINGOPTION_ON;

mfx_EncToolsConf.AdaptiveB = MFX_CODINGOPTION_ON;

mfx_EncToolsConf.SceneChange = MFX_CODINGOPTION_ON;

mfx_EncToolsConf.AdaptivePyramidQuantP = MFX_CODINGOPTION_ON;

mfx_EncToolsConf.AdaptivePyramidQuantB = MFX_CODINGOPTION_ON;

mfx_EncToolsConf.AdaptiveQuantMatrices = MFX_CODINGOPTION_ON;

mfx_EncToolsConf.AdaptiveMBQP = MFX_CODINGOPTION_ON;

mfx_EncToolsConf.AdaptiveLTR = MFX_CODINGOPTION_ON;

mfx_EncToolsConf.AdaptiveRefP = MFX_CODINGOPTION_ON;

mfx_EncToolsConf.AdaptiveRefB = MFX_CODINGOPTION_ON;

mfx_EncToolsConf.BRCBufferHints = MFX_CODINGOPTION_ON;

mfx_EncToolsConf.BRC = MFX_CODINGOPTION_ON;

mfx_ENC_ExtendedBuffers.push_back(

(mfxExtBuffer *)&mfx_EncToolsConf);

[mkpatel3]

Linux was quick checkout on the available system with ffmpeg and EncTools with OneVPL for any fundamental flaw. Larger LA /Look ahead requires more surfaces to be allocated in EncTools, less free memory overall and easy to reproduce issue. Do you monitor available/free memory to see what is going on as current hypothesis from bitstream analysis is repeated frames passed to Encode when issue start to happen. 

[Max_Gud]

If the problem is a lack of memory, how can I increase the amount of memory allocated? I pass it to the encoder
The MaxLength bitstream that exceeds the bitrate size by at least 100 times, i.e. if 8000 bitrates are set, then the MaxLength bitstream is transmitted to 800000, and when using LA this value can be even greater, of course, the value for MaxLength is converted to bits in accordance with the documentation, i.e. the value for MaxLength is set to bitrate * 1000 * 100. For encoding, an internal memory allocation mechanism is used and I get the Surface using the GetSurfaceForEncode function.
That is, I do not directly affect the allocation of memory for the encoding process.
How do I increase the amount of memory allocated by the oneVPL internal allocation mechanism?

For the encoding process with a bitrate of 8000 and LA 100, about 600 megabytes of RAM and about the same amount of video memory are used, this value may change over time, but returns to values close to those indicated above, i.e. memory is being cleaned up properly. The problem occurs only when MFX_EXTBUFF_ENCTOOLS_CONFIG is connected and only after some time, while the values of memory consumed do not differ. Also, I probably should have mentioned this earlier, the more difficult the video is to encode, in terms of the number of objects by type of grass, randomly flickering objects and the like, the faster the problem is identified. With the video that I sent you earlier, the problem is detected at 40-60 minutes of encoding, but with another video that is easier to encode, the problem will be revealed only after 90-100 minutes, while if you encode a static image, the problem may appear either after the same 90-100 minutes, as if some trigger is triggered. Perhaps there is some kind of error and the timestamp gets lost?
I want to pay attention once again, without connecting MFX_EXTBUFF_ENCTOOLS_CONFIG everything works without problems, encoding can take hours without a single drop of frames

[mkpatel3]

We don't recommend using MFX_EXTBUFF_ENCTOOLS_CONFIG which is a private API, exact same results can be obtained for ExtBRC ON and LA > 0. MFX_EXTBUFF_ENCTOOLS_CONFIG is automatically attached internally.  Can you please try this?

[Max_Gud]

Yes, I know that you do not recommend connecting MFX_EXTBUFF_ENCTOOLS_CONFIG, but the output video, when connecting MFX_EXTBUFF_ENCTOOLS_CONFIG, turns out much better quality. I attached screenshots of the same moment in the video, but one video was encoded with MFX_EXTBUFF_ENCTOOLS_CONFIG, and the second without it. By comparing the screenshots, you will be able to understand how much there is a big difference in quality. The encoding parameters are set exactly the same. The result of encoding with MFX_EXTBUFF_ENCTOOLS_CONFIG connected is better in quality than the x264 codec with the slow preset

[mkpatel3]

Just want to make sure Without EncTools case snapshot, you still had LA 40 and EXTBRC on (this will use internal configs)?

[Max_Gud]

I would really like the quality of the output video without the MFX_EXTBUFF_ENCTOOLS_CONFIG connection to be the same as with the connected one, but this does not happen if we use your recommendations.

Perhaps this is somehow related to the problem I described in the second topic - installing ScenarioInfo GAME_STREAMING with LA > 0 causes a NULL PTR or UNSUPPORTED PARAMETERS error and perhaps connecting MFX_EXTBUFF_ENCTOOLS_CONFIG somehow bypasses this problem. Link to the second topic - Re: Enabling EncTools h264 in oneVPL - Intel Community

[Max_Gud]

Update: After installing the latest 4502 driver (WHQL), connecting MFX_EXTBUFF_ENCTOOLS_CONFIG with setting LA > 0 with AVC codec now causes the driver to hang with driver restart without encoder errors.

[JaideepK_Intel]

Hi,

I hope you are doing well.

We got an update from the engineering team like below.

We don’t support direct use of EncTools, it is a part of our internal architecture, and we don't provide direct access to it.

• mfxenctools.h is not public. It’s under api\vpl\private, so we just haven’t expected external customers to access it directly, and are not prepared to address associated issues.
• You can access it by using encoding features that are implemented using EncTools like LookAhead or AGOP.

Note also that we don’t guarantee support for EncTools on Windows.

Can we go ahead and close this case?

Thanks,

Jaideep

[JaideepK_Intel]

Hi,

Can we go ahead and close this case?

Thanks,

Jaideep

[Max_Gud]

Please wait. Yesterday I managed to launch EncTools without using an external buffer, the reason why I could not enable it earlier was in the ScenarioInfo parameter, when installing it, EncTools cannot be activated. But despite the fact that I managed to run EncTools without an external buffer, I again ran into the same problems that were discussed earlier - in modes using Lookahead, I get excellent output video quality, but the frames start dropping after 30-120 minutes of recording.
Since now I don't use a private API, can they help me with this problem? Earlier I was told that the problem may be with a lack of memory, but how to solve this problem?

[JaideepK_Intel]

Hi,

We got a reply from the engineering team as below.

>>They have tried to reproduce the issue with ffmpeg on Windows and Linux and they do not see it.

ffmpeg -c:v av1_qsv -extra_hw_frames 40 -an -stream_loop 135 -i input_2023-05-22_20-49-02.ivf -c:v h264_qsv -preset 1 -async_depth 2 -b:v 8000000 -maxrate 8000000 -minrate 8000000 -bitrate_limit 0 -bufsize 16000000 -rc_init_occupancy 8000000 -low_power true -look_ahead_depth 40 -extbrc 1 -b_strategy 1 -adaptive_i 0 -adaptive_b 0 -bf 7 -g 180 -strict 1 -r 60 -c:a copy avc_ffmpeg_8mbps_g180r8la40_hrd1_loop135.mp4

Sorry to say this, but they were unable to reproduce your issue from their end.

Can we go ahead and close this case?

Thank's

Jaideep

[JaideepK_Intel]

Hi,

Can we go ahead and close this case ?

Thanks,

Jaideep

[Max_Gud]

I was able to reproduce the problem using FFMPEG.

The following parameters were used:

-extra_hw_frames 40 -preset 1 -async_depth 2 -bitrate_limit 0 -low_power true -look_ahead_depth 40 -extbrc 1 -b_strategy 1 -adaptive_i 0 -adaptive_b 0 -bf 7 -strict 1 -r 60 -rdo 1 -mbbrc 1 -b_strategy 1 -idr_interval 0 -look_ahead 1 -look_ahead_downsampling off -profile high

In order to reproduce the problem during the encoding process, it is enough to create a small load on the CPU, in my case, it was the playback of the video in the AV1 codec using processor power, the load on the processor increased approximately by 15%, at the beginning of playback, frame drops began almost immediately and after closing the video, the drops did not stop.

[mkpatel3]

Can you please paste the full ffmpeg command line? As incompatible parameters could cause other side effects and delay repro of the issue. 

[mkpatel3]

This is what I tried and still no repro.. I also had CPU loaded with ivf playback on the side ~20%. 

ffmpeg-6.0-full_build\bin>
ffmpeg -c:v av1_qsv -extra_hw_frames 40 -an -stream_loop 10 -i input_2023-05-22_20-49-02.ivf -c:v h264_qsv -preset 1 -async_depth 2 -b:v 8000000 -maxrate 8000000 -minrate 8000000 -bitrate_limit 0 -bufsize 16000000 -rc_init_occupancy 8000000 -low_power true -look_ahead_depth 40 -extbrc 1 -b_strategy 1 -adaptive_i 0 -adaptive_b 0 -bf 7 -g 180 -strict 1 -r 60 -rdo 1 -mbbrc 1 -look_ahead 1 -look_ahead_downsampling off -c:a copy avc_ffmpeg_8mbps_g180r8la40_hrd1_loop10_rdo1_mbbrc1_la1_other2.mp4

[Max_Gud]

The scenario in which I use an encoder is realtime video recording, real-time broadcast, most likely you will not be able to reproduce using video transcoding

[Max_Gud]

ffmpeg -f dshow -use_wallclock_as_timestamps 0 -pixel_format nv12 -rtbufsize 500M -thread_queue_size 8192 -framerate 60 -i video="@capture_card_device_id" -c:v h264_qsv -async_depth 2 -bitrate_limit 0 -low_power true -look_ahead_depth 40 -extbrc 1 -b_strategy 1 -adaptive_i 0 -adaptive_b 0 -bf 7 -strict 1 -r 60 -rdo 1 -mbbrc 1 -b_strategy 1 -idr_interval 0 -look_ahead 1 -look_ahead_downsampling off -bufsize 16M -maxrate 8M -minrate 8M -rc_init_occupancy 8M -profile:v high -b:v 8M -c:a copy "test.mp4"

Here is an approximate code that works approximately like OBS-Studio, but even with it, if you cause a load on the processor during recording, duplicate frames appear. I want to clarify that the CPU load does not exceed 60%.

When using a plugin or custom output using FFMPEG h264_qsv in OBS-Studio, the situation is much worse than in the example given.

[Max_Gud]

I also attach a sample with the reproduction of the problem using FFMPEG

[JaideepK_Intel]

Hi,

I hope you are good.

Sorry for the delay. The command provided by mkpatel is working for us, but we are facing issues with the below command after adding flag -profile:v high.

Attaching a screenshot for reference.

ffmpeg -c:v av1 -extra_hw_frames 40 -an -stream_loop 10 -i "2023-05-22 20-49-02_new.mkv" -c:v h264 -extra_hw_frames 40 -preset 1 -async_depth 2 -bitrate_limit 0 -low_power true -look_ahead_depth 40 -extbrc 1 -b_strategy 1 -adaptive_i 0 -adaptive_b 0 -bf 7 -strict 1 -r 60 -rdo 1 -mbbrc 1 -b_strategy 1 -idr_interval 0 -look_ahead 1 -look_ahead_downsampling off -profile high -c:a copy cpy.mp4

Is there anything we are missing here?

Thanks,

Jaideep

[Max_Gud]

Try changing -profile high to -profile:v high and change -c:v h264 on -c:v h264_qsv, also you are trying to allocate -extra_hw_frames for the AV1 software codec, change -c:v av1 on -c:v av1_qsv or try to remove -extra_hw_frames for it

[JaideepK_Intel]

Hi,

I hope you are doing well. We are unable to reproduce your issue from our end, we are working on this internally. We will get back to you with an update.

Thanks,

Jaideep