Intel® Distribution of OpenVINO™ Toolkit
Community assistance about the Intel® Distribution of OpenVINO™ toolkit, OpenCV, and all aspects of computer vision-related on Intel® platforms.

Openvino threading question

k__alex2
Beginner
1,526 Views

Hi everyone and happy new year!

I have a question related to Openvino.

is it possible to create N threads (using for example pthreads) and inside each thread have an instance of Openvino network?

Right now (if I'm understanding correctly) to use multiple threads (although inside openvino) is to use PluginConfigParams::KEY_CPU_THREADS_NUM when loading the network.

If it's not possible to have N threads would it be possible to have N processes? I tested running N benchmark_app processes, each using one thread (as a cli argument) but I ended up utilizing one core for all processes. I don't know if some environment variable would solve this.

 

Thanks!

0 Kudos
5 Replies
Monique_J_Intel
Employee
1,526 Views

Hi Alex,

Have you tried to set OMP_NUM_THREADS environment variable? If not, you should give that a try it's worked for me.

Kind Regards,

Monique Jones

k__alex2
Beginner
1,526 Views

Hi Monique,

Thanks for the answer. I have used OMP_NUM_THREADS but ended up not using it..

Here's my problem, if you have N instances of OpenVino and set OMP_NUM_THREADS to N, you end up with N^2 threads, which is something I don't want..

If you have N instances of OpenVino and set OMP_NUM_THREADS to 1, then all instances share the single core, which again is not something I want.

I ended up setting KEY_CPU_BIND_THREAD to NO and setting KEY_CPU_THREADS_NUM to 1. So I can have N OpenVino instances and each instance can use one core. I know it's a crappy solution, I'm not proud of it, but I couldn't find any alternative, if there is however please let me know. Right now, I can't use (due to higher bias) batchsize >1.

nikos1
Valued Contributor I
1,526 Views

Hi Alex,

> is it possible to create N threads (using for example pthreads) and inside each thread have an instance of Openvino network?

It is possible, but may depend on the OS and inference device. I am running multiple OpenVino instances from various threads + different networks. Are you running on Linux-CPU device? In one of my pipelines, on Windows, I hit some threading issues and had to mutex around them. What issues are you seeing?

Cheers,

Nikos

k__alex2
Beginner
1,526 Views

Γεια σου νικο :) ,

The main issue is, how do you go about pinning each Openvino instance's thread to a separate core (assuming each openvino instance has a single thread for inference).

Because right now, I have N threads, and each thread has it's own OpenVino instance, each Openvino creates one thread as well (due to setting KEY_CPU_THREADS_NUM to 1 in LoadNetwork config), and it works ok, other than that, instead of saturating N cores, I see utilization across all cores, that's why I want to pin the threads on cores (apart from any affinity related benefits).

 

nikos1
Valued Contributor I
1,526 Views

:-)

Ahhh okay I understand .. cannot think of any other alternatives unless you modify the SDK source code.

cheers

nikos

Reply