Alternating 2 I.E.

[bell__sefi]

I am trying to run inference using 2 different models on the same machine (hopefully using all CPU resources for each model).

I understand I can't have 2 IECores running together.

Can I, instead, convert to OpenVino in advance and just load the checkpoint when inference is necessary? 

How can I verify the unnecessary IECore is down? Is there a method to do remove it?

[Iffa_Intel]

Greetings,

Openvino does support multi-device execution. A plugin called Multi-Device plugin, automatically assigns inference requests to available computational devices to execute the requests in parallel

You can refer here:

https://docs.openvinotoolkit.org/latest/openvino_docs_IE_DG_supported_plugins_MULTI.html

with multi-device the application logic left unchanged, so you don't need to explicitly load the network to every device, create and balance the inference requests and so on. From the application point of view, this is just another device that handles the actual machinery.

The only thing that is required to leverage performance is to provide the multi-device (and hence the underlying devices) with enough inference requests to crunch. For example if you were processing 4 cameras on the CPU (with 4 inference requests), you may now want to process more cameras (with more requests in flight) to keep CPU+GPU busy via multi-device.

Sincerely,

Iffa

[bell__sefi]

I guess I didn't explain my self well. 

I am interested in holding 2 openvino-converted models (= IECores) on the same device (e.g. AWS's C5.4xlarge). Then, when a batch of inputs arrives, I would like to pick which of the 2 to use, but I would like to avoid the time it takes to load the checkpoint into the IECore object.

Can that be done somehow?

[Sahira_Intel · Accepted solution]

Hi,

I tried looking for some demos or samples that did this, but was not able to find any. In terms of choosing which of the 2 inputs to use, I think the best way to achieve this would be to run some script that dealt with the inputs. But I don't think it would be possible to avoid loading the checkpoint into the IECore object. 

I hope this information is helpful. Please let me know if you have any further questions.

Sincerely,

Sahira