Enabling EncTools h264 in oneVPL

[Max_Gud]

In order not to create a new topic, can I ask one more question?
The question concerns the CO3.ScenarioInfo parameter.
When researching the code in the oneVPL-gpu repository, I found that setting SCENARIO_GAME_STREAMING and Lookahead depth > 0 is acceptable for the AV1 codec, this should include additional frame processing and give a better image. But I ran into the problem that when installing Scenario GAME_STREAMING and LA > 0, I get the MFX error NULL_PTR (-3) both with the AV1 codec and with any other codec, could you help me figure out this issue?

[DiyaN_Intel]

Hi, 

Thank you for posting in Intel Communities.

Could you please share the following details- 

1. Sample reproducer code.

2. oneVPL version.

3. Exact steps and the commands used.

4. OS and Processor details.

Thanks and Regards,

Diya

[Max_Gud]

Hi, I am using oneVPL dispatcher version 2.9 with the latest beta drivers 4382.

Windows 10 and Windows 11 are used.

The GPUs used are UHD 770 and Arc A380.

Processor 12700K.

I put together a small project from hello-encode in order to show the error, the project is attached to the message.

There is an assembled file in the project encode.exe and the test.i420 and test.nv12 files, if you start the encoding process with the parameters -i test.i420 -w 320 -h 240 -sw, the encoding process goes without problems, but this is a software implementation of the encoder, and if you start the process with the parameters -i test.nv12 -w 320 -h 240 -hw we get the error "Encode init failed" (If you check the error code, it will show the status -2 (MFX_ERR_NULL_PTR)). With AVC and AV1 codecs, the error occurs both in SW and HW mode. If you change MFX_SCENARIO_GAME_STREAMING to MFX_SCENARIO_LIVE_STREAMING, the encoding passes without errors.

In the code you can see that I have set the parameters mfx_CO2.Lookahead = 60 and mfx_CO3.ScenarioInfo = MFX_SCENARIO_GAME_STREAMING

[Max_Gud]

With the latest beta driver 4499, the initialization error code has changed to -15 (MFX_ERR_INVALID_VIDEO_PARAM) for the HW implementation, for the HEVC codec, nothing has changed for the SW implementation, it works without problems.

[DiyaN_Intel]

Hi, 

We are investigating your issue at our end and working on this internally. We will get back to you with an update soon.

Regards, 

Diya

[DiyaN_Intel]

Hi, 

Good day to you.

While investigating the issue from our end we have observed an issue running the original hello-encode program present in the GitHub repository(https://github.com/oneapi-src/oneVPL).

Our dev team is looking into it.

Please confirm whether you have observed any issues while running the hello-encode sample without any modifications.

Thanks and Regards,

Diya

[Max_Gud]

Hello, to be honest, I do not know, because I have not tried to run sample without modifications.

[DiyaN_Intel]

Hi,

Please try to run the hello_encode program without any modifications from this GitHub repository https://github.com/oneapi-src/oneVPL and let us know if you face any challenges, then we can investigate further.

Regards,

Diya

[Max_Gud]

I don't quite understand how the presented case is related to the problem for which this topic was opened? I have provided a ready-made project for testing, with the source code and a compiled .exe file. The presented project differs from the fact that the oneVPL repository differs only in that lines with the setting of Lookahead and ScenarioInfo parameters have been added, as well as the dispatcher and API files have been moved to the same folder for connection.

In addition, the assembly of the example is impossible without modification, because the file placement paths of the dispatcher and the oneVPL API specified in the example are not valid, you will have to move the files to build in any case. Also, to build an example, you will have to create a project manually, because there are no .cmake files for examples in the repository. In addition to the above, the project will not be assembled without configuring the compiler with ignoring Warning 4996 due to the mfxvideo++.h and main.cpp file, which uses functions that are declared deprecated.

[DiyaN_Intel]

Hi, 

We are investigating your issue at our end and working on this internally. We will get back to you with an update soon.

Regards, 

Diya

[Denis-Volkov_Intel]

Hello Max_Gud,

As I see, you've made a several one-by-one experiments affected 3 encoders (AVC, HEVC, AV1) with getting a different cross-codec results... let's try to resolve your questions step-by-step also.

For beginning, let's start from the primary topic:
>> this should include additional frame processing and give a better image

As I right understand from the context - your primary purpose is to get a better quality encoding using existed VPL API/possibilities.
Below I share my suggestions and thoughts about that (also based on your "hello-encode" sample code) - even maybe using LookAhead and MFX_SCENARIO_GAME_STREAMING will not be the best solution for you:

• first of all, I suggest to use TargetUsage = 1 (MFX_TARGETUSAGE_BEST_QUALITY or MFX_TARGETUSAGE_1) instead of MFX_TARGETUSAGE_BALANCED in "hello-encode" example
  • TargetUsage = 1 introduced the best frame-oriented quality (by increasing Frame encoding complexity)
• you can try to use ICQ RateControl (MFX_RATECONTROL_ICQ) instead of VBR or CBR - and regulate the quality via mfxInfoMFX::ICQQuality parameter (from 1 to 51; value 10-16 some kind of optimal for better Quality)

I also have some questions about your encoding requirement and expectation:

• what kind of video content do you want to use? Is it some regular or even mix/different content or maybe some specific videos (like Game content, Films, Clips with lot of scene changes, ... ) ?
• do you want to encode the streams for local storage or using network/live-streaming ?
  • I'm asking to understand Bitrate and Delay requirements
• which encoder do you prefer - AVC / HEVC / AV1 - or you will decide based on experiments or usage (so, every encoder is needed with the similar priority) ?

This information will be helpful to provide some more useful ideas and concrete solutions from my side.

P.S.: I will reply about LookAhead and MFX_SCENARIO_GAME_STREAMING via a separate comments later.

Best regards,
Denis

[Max_Gud]

Hello Denis, thank you for your recommendations, I want to explain the parameters that I use:

- I use MFX_TARGETUSAGE_1 by default, but in the example that was sent for consideration, I did not change TargetUsage.

- The intended purpose for which oneVPL is used is real-time broadcasting, as well as for local recording. For local recording, I use RateControl CQP, but for online broadcasts on advanced platforms (Twitch, Youtube, Trovo and others), the main RateControl is CBR, so it is used.

- The main content of recordings and online broadcasts is game content, to a lesser extent IRL content (recording from a webcam when communicating with a chat).

- At the moment, the main codec for most streaming platforms is AVC, but either HEVC or AV1 is used for broadcasting on Youtube. For local recording, HEVC or AV1 is also used.

[Denis-Volkov_Intel]

Thank you very much for sharing this information - It's very helpful for understanding your usage models!

[Denis-Volkov_Intel]

Now about MFX_SCENARIO_GAME_STREAMING and LookAhead:

• MFX_SCENARIO_GAME_STREAMING is implemented specially for Game content encoding (and live streaming for that), it's include some frame-encoding-tunning which introduce subjective quality improvement for Game content superficially - and it's not necessary that these benefits will be applied for another type of content (that's why I asked about your encoding requirements in my previous comment).
• LoodAhead - special encoding mode where encoder performs pre analysis of several frames before actual encoding starts - this mode especially useful for scene-change content (by early detection and BRC methods adaptation), but increase the delays and additional Performance decreasing (so again, not sure that it will be helpful for your scenarios).

Now back to errors/issues which your described - "mfx_CO2.Lookahead = 60 and mfx_CO3.ScenarioInfo = MFX_SCENARIO_GAME_STREAMING":

• I suggest to set LowPower=ON directly - "encodeParams.mfx.LowPower = MFX_CODINGOPTION_ON"
• you can try to use ExtBRC - "mfx_CO2.ExtBRC = MFX_CODINGOPTION_ON"

Please let us know about the results of experiment on you side - especially if I will try different Encoders (my suggestion are primary for AVC as a basic one).

Best regards,
Denis

[Max_Gud]

I tried using CO2.ExtBRC, but I have a problem with it. When installing CO2.ExtBRC and Lookahead > 0 (20, 30, 40, 60, 100 and etc.), I get a working Lookahead and an output video of excellent quality, but after some time of recording or broadcasting, frame drops (or duplication of I frames) begin, the number of which constantly increases until the video is brought into a slideshow with one frame displayed every few seconds. I have created a topic on the problem, which you can read at this link - Enabling EncTools h264 in oneVPL - Intel Community

[Denis-Volkov_Intel]

Hello Max_Gud,
Thank you for your reply and sharing the thread about ExtBRC - I analyze all the information there and it's clear to me now!

So, lets return back to HW LookAhead (without ExtBRC).

As you mentioned previously, you have 2 types of Intel HW:

• 12700K - which is Alder Lake (TGLx media) - integrated (https://www.intel.com/content/www/us/en/products/sku/134594/intel-core-i712700k-processor-25m-cache-up-to-5-00-ghz/specifications.html )
• Arc A380 - which is Alchemist/DG2 media - discrete (https://www.intel.com/content/www/us/en/products/sku/227959/intel-arc-a380-graphics/specifications.html )

I suggest to start from 12700K to make LoodAhead workable here first for AVC and HEVC (as some kind of reference) and then switch to Arc with adding AV1 too (as AV1 encoder is supported started from Arc/DG2 - https://github.com/intel/media-driver#decodingencoding-features ).

I've made some experiments on my local TGL (i7-1185G7) machine with the similar Media as for 12700K ADL - on my side LookAhead is working for AVC and HEVC both on our internal testing tool and using your sample.

I also modified your sample by adding "encodeParams.mfx.LowPower = MFX_CODINGOPTION_ON;" - after that LookAhead is working both for AVC and HEVC (I changed Codec and recompiled the sample for experiments).

So, minimal requirement for enabling LookAhead on TGLx media:

• set LowPower = ON
• choose AVC or HEVC encoder
• RateControlMode CBR or VBR
• Set LookAhead value in range [10, 100] and ScenarioInfo = MFX_SCENARIO_GAME_STREAMING
• use HW implementation
• setting width/height and appropriate TargetKbps
• you can additionally set GOP structure (GopPicSize / GopRefDist) or it will be set by default

Some additional suggestions from my side:

• MediaSDK/VPL could set some parameters to default value in case if they are not set by User (usually "0"), and also update/change some parameters in case of incompatibility with others
  • that's mean that MFXVideoENCODE_Init function can return a warning MFX_WRN_INCOMPATIBLE_VIDEO_PARAM (> 0) instead of MFX_ERR_NONE (== 0) - https://github.com/Intel-Media-SDK/MediaSDK/blob/master/doc/mediasdk-man.md#mfxvideoencode_init 
• I also recommend you to use function  MFXVideoENCODE_GetVideoParam after Init to see which parameters has been changed during Initialization (especially in case of MFX_WRN_INCOMPATIBLE_VIDEO_PARAM status) - https://github.com/Intel-Media-SDK/MediaSDK/blob/master/doc/mediasdk-man.md#mfxvideoencode_getvideoparam 

Max_Gud - please help to check LoodAhead + MFX_SCENARIO_GAME_STREAMING on 12700K first - please help to share the results of your experiments especially in case of any issues.

I also share "hello-encode.cpp" from your sample updated on my side.

[Max_Gud]

I tried your example with the file you provided, but as a result I get a -2 error.
I tried running your example on both UHD 770 and Arc A380, the result is the same.

[Denis-Volkov_Intel]

Thank you for your feedback!

Error "-2" is MFX_ERR_NULL_PTR (https://github.com/Intel-Media-SDK/MediaSDK/blob/master/api/include/mfxdefs.h#L197 ) - as it's return from Init(...) - usual case as there is missing some parameters like incorrect input stream name, resolution, implementation, adapter (I see DeviceIDs here + HW impl - so it's not in this case)...

• I do not see resolution of the stream in your command-line - did you "move" inside the sample implementation?
• one more question - did you try to remove "LookAhead" and "MFX_SCENARIO_GAME_STREAMING" as you done in the previous experiments?
  • Simply say, is this error "-2" reproduced only with LookAhead / MFX_SCENARIO_GAME_STREAMING and Encoding works correct without these settings?

HD 770 / DeviceID = 0x4680 is Alder Lake (TGLx media) - similar is mine Tiger Lake environment...

[Max_Gud]

I set the resolution parameters inside so as not to specify each time

If you remove MFX_SCENARIO_GAME_STREAMING, then everything starts working. If you leave MFX_SCENARIO_GAME_STREAMING and remove lookahead, then the error remains

[Denis-Volkov_Intel]

Got it - thank you very much for your feedback!

We will try to use Alder Lake or/and Arc machines directly to reproduce your results

[DiyaN_Intel]

Hi ,

Can you please share the error logs for VPL and UMD ?

Please help us with traces for analysis purpose as well.

Regards,

Diya