Openvino threading question

[k__alex2]

Hi everyone and happy new year!

I have a question related to Openvino.

is it possible to create N threads (using for example pthreads) and inside each thread have an instance of Openvino network?

Right now (if I'm understanding correctly) to use multiple threads (although inside openvino) is to use PluginConfigParams::KEY_CPU_THREADS_NUM when loading the network.

If it's not possible to have N threads would it be possible to have N processes? I tested running N benchmark_app processes, each using one thread (as a cli argument) but I ended up utilizing one core for all processes. I don't know if some environment variable would solve this.

Thanks!

[Monique_J_Intel]

Hi Alex,

Have you tried to set OMP_NUM_THREADS environment variable? If not, you should give that a try it's worked for me.

Kind Regards,

Monique Jones

[k__alex2]

Hi Monique,

Thanks for the answer. I have used OMP_NUM_THREADS but ended up not using it..

Here's my problem, if you have N instances of OpenVino and set OMP_NUM_THREADS to N, you end up with N^2 threads, which is something I don't want..

If you have N instances of OpenVino and set OMP_NUM_THREADS to 1, then all instances share the single core, which again is not something I want.

I ended up setting KEY_CPU_BIND_THREAD to NO and setting KEY_CPU_THREADS_NUM to 1. So I can have N OpenVino instances and each instance can use one core. I know it's a crappy solution, I'm not proud of it, but I couldn't find any alternative, if there is however please let me know. Right now, I can't use (due to higher bias) batchsize >1.

[nikos1]

Hi Alex,

> is it possible to create N threads (using for example pthreads) and inside each thread have an instance of Openvino network?

It is possible, but may depend on the OS and inference device. I am running multiple OpenVino instances from various threads + different networks. Are you running on Linux-CPU device? In one of my pipelines, on Windows, I hit some threading issues and had to mutex around them. What issues are you seeing?

Cheers,

Nikos

[k__alex2]

Γεια σου νικο :) ,

The main issue is, how do you go about pinning each Openvino instance's thread to a separate core (assuming each openvino instance has a single thread for inference).

Because right now, I have N threads, and each thread has it's own OpenVino instance, each Openvino creates one thread as well (due to setting KEY_CPU_THREADS_NUM to 1 in LoadNetwork config), and it works ok, other than that, instead of saturating N cores, I see utilization across all cores, that's why I want to pin the threads on cores (apart from any affinity related benefits).

[nikos1]

:-)

Ahhh okay I understand .. cannot think of any other alternatives unless you modify the SDK source code.

cheers

nikos