Intel® Distribution of OpenVINO™ Toolkit
Community support and discussions about the Intel® Distribution of OpenVINO™ toolkit, OpenCV, and all things computer vision-related on Intel® platforms.

Alternating 2 I.E.

bell__sefi
Beginner
250 Views

I am trying to run inference using 2 different models on the same machine (hopefully using all CPU resources for each model).

I understand I can't have 2 IECores running together.

Can I, instead, convert to OpenVino in advance and just load the checkpoint when inference is necessary? 

How can I verify the unnecessary IECore is down? Is there a method to do remove it?

0 Kudos
1 Solution
Sahira_Intel
Moderator
204 Views

Hi,

I tried looking for some demos or samples that did this, but was not able to find any. In terms of choosing which of the 2 inputs to use, I think the best way to achieve this would be to run some script that dealt with the inputs. But I don't think it would be possible to avoid loading the checkpoint into the IECore object. 

I hope this information is helpful. Please let me know if you have any further questions.

Sincerely,

Sahira 

View solution in original post

3 Replies
Iffa_Intel
Moderator
231 Views

Greetings,


Openvino does support multi-device execution. A plugin called Multi-Device plugin, automatically assigns inference requests to available computational devices to execute the requests in parallel


You can refer here:

https://docs.openvinotoolkit.org/latest/openvino_docs_IE_DG_supported_plugins_MULTI.html


with multi-device the application logic left unchanged, so you don't need to explicitly load the network to every device, create and balance the inference requests and so on. From the application point of view, this is just another device that handles the actual machinery.


The only thing that is required to leverage performance is to provide the multi-device (and hence the underlying devices) with enough inference requests to crunch. For example if you were processing 4 cameras on the CPU (with 4 inference requests), you may now want to process more cameras (with more requests in flight) to keep CPU+GPU busy via multi-device.


Sincerely,

Iffa


bell__sefi
Beginner
225 Views

I guess I didn't explain my self well. 

I am interested in holding 2 openvino-converted models (= IECores) on the same device (e.g. AWS's C5.4xlarge). Then, when a batch of inputs arrives, I would like to pick which of the 2 to use, but I would like to avoid the time it takes to load the checkpoint into the IECore object.

Can that be done somehow?

 

Sahira_Intel
Moderator
205 Views

Hi,

I tried looking for some demos or samples that did this, but was not able to find any. In terms of choosing which of the 2 inputs to use, I think the best way to achieve this would be to run some script that dealt with the inputs. But I don't think it would be possible to avoid loading the checkpoint into the IECore object. 

I hope this information is helpful. Please let me know if you have any further questions.

Sincerely,

Sahira 

View solution in original post

Reply