Loading a big model on 2 or more NCS2

[Gama]

I'm trying to load a big model 1.3 GB on 2 sticks but I got an error. is it even supported to run big models on multiple devices ? 

[Aznie_Intel]

Hi Gama,

Thanks for reaching out.

The internal memory for Neural Compute Stick 2 (NCS 2) is 500MB. Loading a network in multiple devices is known as heterogeneous execution in OpenVINO.

The heterogeneous plugin enables computing for inference on one network on several devices. Purposes to execute networks in the heterogeneous mode are as follows:

• To utilize accelerators power and calculate heaviest parts of the network on the accelerator and execute not supported layers on fallback devices like CPU
• To utilize all available hardware more efficiently during one inference

Refer to Heterogeneous Execution.

Regards,

Aznie

[Gama]

First of all I was able to load 10 models of size 91.5 MB on a single stick how is that possible with the size 500 MB memory ?
second, I have to load the model first before defining the heterogenous or any other mode of inference so if I get an error loading models larger than the memory I will not reach to the step of defining it ?

[Aznie_Intel]

Hi Gama,

What error are you seeing at first as you mentioned before?

Regards,

Aznie

[Gama]

after using 

core = IECore()
model_inst = core.read_network(model= "face_model.xml", weights="face_model.bin")
device = "MYRIAD"

model0 = core.load_network(network=model_inst, device_name = device)

the same ........... until model19 

I got

failed to allocate graph: MYRIAD device is not opened.

after loading the 20 models and got this at model 21 

and How exactly should I load the models to use heterogeneous execution

[Aznie_Intel]

Hi Gama,

Heterogenous plugin means that during the loading of the network to the Heterogeneous plugin, the network is divided into separate parts and loaded to dedicated plugins. Intermediate blobs between these subgraphs are allocated automatically in the most efficient way.

However, in your case, the issue is the NCS2 is not working. Can you try running your model using Benchmark_app with MYRIAD plugin.

Benchmark_app -m model.xml -d MYRIAD

Meanwhile, please refer to this OpenVINO API Tutorial to learn how to load the IR model. Also, please note that Intel® Neural Compute Stick 2 (NCS2) is not supported in the latest release, OpenVINO 2022.3.1, please use OpenVINO 2022.1.

Regards,

Aznie

[Gama]

I loaded the models using HETERO but I couldn't load more than 20 models of size 91.5 MB on 2 sticks. 

[Aznie_Intel]

Hi Gama,

What error did you get? NCS 2 will be unable to handle 20 models with the of 90Mb. 20 models of 90Mb will make the size of 1.8Gb. The internal memory of Intel® NCS2 is about 500 MB and 100 MB of the memory is set aside for Intermediate Representation (IR) processing.

Regards,

Aznie

[Gama]

I got the same as before

failed to allocate graph: MYRIAD device is not opened.

but after loading 20 models (when trying to load model number 21) I don't know how there is no error when creating 20 instance from loading the model and why after 20 models I got this error.

can you suggest a model that I can use to make that test ? 

[Aznie_Intel]

Hi Gama,

Have you tried running your model using Benchmark_app with MYRIAD plugin as I asked before? Please try using one of your models to check whether your NCS2 is working or not. Also, please note that it is impossible to load all 20 models since the size is too large for NCS2 to handle.

You may check OpenVINO™ Toolkit Intel’s Pre-Trained Models that you can use for learning and demo purposes or for developing deep learning software.

Regards,

Aznie

[Aznie_Intel]

Hi Gama,

Thank you for your question. If you need any additional information from Intel, please submit a new question as this thread is no longer being monitored.

Regards,

Aznie