How to limit the iGPU/CPU usage

xxbird · ‎07-10-2023

Hi!

I utilize OpenVINO in my prototype. When doing inference on iGPU, the OpenVINO seems to occupy all iGPU compute units and even the CPU occupation raises to >100%. This indicates severe performance degradation for the other programs running on the prototype.

So I am wondering if there is a way to limit or set CPU/iGPU usage.

Thanks,

X. X.

Megat_Intel · ‎07-11-2023

Hi X. X.,

Thank you for reaching out to us.

To limit CPU usage, you can configure to limit the number of threads (-nthreads) to use for inference on the CPU. While for GPU, you can configure to limit the number of streams (-nstreams) for inference on the GPU devices. You can adjust the streams and threads accordingly to obtain the best results. You can refer to the CPU Optimization Guide for more information.

For your information, I validated on my end by running the OpenVINO™ C++ Benchmark Tool. I ran the Benchmark tool on the CPU using Auto configuration and then configured the -nthreads to 2. Next, I ran the Benchmark tool on the GPU with Auto configuration and then configured the -nstreams to 1. I share my results below:

CPU Auto

CPU -nthreads 2

GPU Auto

GPU -nstreams 1

Please note that limiting both the -nstreams and -nthreads will affect the performance results. Below are the performance results that I obtained:

GPU

[Step 11/11] Dumping statistics report

[ INFO ] Execution Devices: [ GPU.0 ]

[ INFO ] Count: 3416 iterations

[ INFO ] Duration: 60114.78 ms

[ INFO ] Latency:

[ INFO ] Median: 65.87 ms

[ INFO ] Average: 70.29 ms

[ INFO ] Min: 54.77 ms

[ INFO ] Max: 169.69 ms

[ INFO ] Throughput: 56.82 FPS

GPU (-nstreams 1)

[Step 11/11] Dumping statistics report

[ INFO ] Execution Devices: [ GPU.0 ]

[ INFO ] Count: 2750 iterations

[ INFO ] Duration: 60031.11 ms

[ INFO ] Latency:

[ INFO ] Median: 20.91 ms

[ INFO ] Average: 21.79 ms

[ INFO ] Min: 17.70 ms

[ INFO ] Max: 73.27 ms

[ INFO ] Throughput: 45.81 FPS

CPU

[Step 11/11] Dumping statistics report

[ INFO ] Execution Devices: [ CPU ]

[ INFO ] Count: 2744 iterations

[ INFO ] Duration: 60128.57 ms

[ INFO ] Latency:

[ INFO ] Median: 81.87 ms

[ INFO ] Average: 87.60 ms

[ INFO ] Min: 67.35 ms

[ INFO ] Max: 251.04 ms

[ INFO ] Throughput: 45.64 FPS

CPU (-nthreads 2)

[Step 11/11] Dumping statistics report

[ INFO ] Execution Devices: [ CPU ]

[ INFO ] Count: 1205 iterations

[ INFO ] Duration: 60060.44 ms

[ INFO ] Latency:

[ INFO ] Median: 50.76 ms

[ INFO ] Average: 49.81 ms

[ INFO ] Min: 28.65 ms

[ INFO ] Max: 84.27 ms

[ INFO ] Throughput: 20.06 FPS

Regards,

Megat

xxbird · ‎07-11-2023

Hi Megat,

Thank you for your reply!

I'll try to configure nstream and nthread as you said. BTW, I saw some comments online saying that modifying GPU_QUEUE_THROTTLE and GPU_QUEUE_PRIORITY might work. I already tried it and it worked in my scenario. Could you tell me what these two parameters mean and how they work?

Thanks again.

Regrads,

X. X

Megat_Intel · ‎07-12-2023

Hi X. X.,

Both the GPU_QUEUE_PRIORITY and GPU_QUEUE_THROTTLE are part of the Global Variables for Intel GPU OpenCL specific properties. Using it will instruct the GPU plugin to use the OpenCL queue (Priority/Throttle) hints.

For your information, the Priority Hints (GPU_QUEUE_PRIORITY is the extension implementation in OpenVINO™) add priority hints for OpenCL. Choosing the High option in OpenVINO™ will indicate the highest priority. However, the priority hints do not specify the scheduling behavior or any minimum guarantees. You can read more in The OpenCL Extension Specification document, chapter 9.18.

On the other hand, the Throttle Hints (GPU_QUEUE_THROTTLE is the extension implementation in OpenVINO™) adds throttle hints for OpenCL. Selecting the High option in OpenVINO™ indicates the highest energy consumption. But, the throttle hints extension does not specify the throttling behavior or any minimum guarantees. More information can be found in The OpenCL Extension Specification document, chapter 9.19.

Do note that the throttle hint is related to the defined priority hint extension. For example, a task may have high priority but at the same time be executed at an optimized throttle setting ( Low Throttle ). You can try to test the variables to get the best optimal results for your application. Hope this helps.

Regards,

Megat

Megat_Intel · ‎07-19-2023

Hi X. X.,

This thread will no longer be monitored since we have provided a suggestion. If you need any additional information from Intel, please submit a new question.

Regards,

Megat