Community
cancel
Showing results for 
Search instead for 
Did you mean: 
Highlighted
Beginner
119 Views

Alternating 2 I.E.

Jump to solution

I am trying to run inference using 2 different models on the same machine (hopefully using all CPU resources for each model).

I understand I can't have 2 IECores running together.

Can I, instead, convert to OpenVino in advance and just load the checkpoint when inference is necessary? 

How can I verify the unnecessary IECore is down? Is there a method to do remove it?

0 Kudos

Accepted Solutions
Highlighted
Moderator
73 Views

Hi,

I tried looking for some demos or samples that did this, but was not able to find any. In terms of choosing which of the 2 inputs to use, I think the best way to achieve this would be to run some script that dealt with the inputs. But I don't think it would be possible to avoid loading the checkpoint into the IECore object. 

I hope this information is helpful. Please let me know if you have any further questions.

Sincerely,

Sahira 

View solution in original post

0 Kudos
3 Replies
Highlighted
Moderator
100 Views

Greetings,


Openvino does support multi-device execution. A plugin called Multi-Device plugin, automatically assigns inference requests to available computational devices to execute the requests in parallel


You can refer here:

https://docs.openvinotoolkit.org/latest/openvino_docs_IE_DG_supported_plugins_MULTI.html


with multi-device the application logic left unchanged, so you don't need to explicitly load the network to every device, create and balance the inference requests and so on. From the application point of view, this is just another device that handles the actual machinery.


The only thing that is required to leverage performance is to provide the multi-device (and hence the underlying devices) with enough inference requests to crunch. For example if you were processing 4 cameras on the CPU (with 4 inference requests), you may now want to process more cameras (with more requests in flight) to keep CPU+GPU busy via multi-device.


Sincerely,

Iffa


0 Kudos
Highlighted
Beginner
94 Views

I guess I didn't explain my self well. 

I am interested in holding 2 openvino-converted models (= IECores) on the same device (e.g. AWS's C5.4xlarge). Then, when a batch of inputs arrives, I would like to pick which of the 2 to use, but I would like to avoid the time it takes to load the checkpoint into the IECore object.

Can that be done somehow?

 

0 Kudos
Highlighted
Moderator
74 Views

Hi,

I tried looking for some demos or samples that did this, but was not able to find any. In terms of choosing which of the 2 inputs to use, I think the best way to achieve this would be to run some script that dealt with the inputs. But I don't think it would be possible to avoid loading the checkpoint into the IECore object. 

I hope this information is helpful. Please let me know if you have any further questions.

Sincerely,

Sahira 

View solution in original post

0 Kudos