Memory usage openvino

[Deblauwe__Tom]

Hello,

I am using openvino to run inferences on the movidius stick 1, with a network that is 40MB in size. To load the network, I need a CNNNetreader which allocates the network. So that then takes the 40MB memory. Then I need to call getNetwork() on that reader to make a copy, so now I have 80MB in memory. I can then delete the CNNNetreader object and I have 40MB again in the CNNNetwork object. Now I need to make the executable network, and then I again have 80MB. After having the executable network, I delete the CNNNetwork object and still have 40MB in memory. This 40MB remains there during inferences. But here is my question: why does the network still need the 40MB, because I am running the inference on the movidius stick, so I would think that once the network is loaded into the stick, I don't need it anymore resident in main memory?

Best regards,

Tom

[David_C_Intel]

Hi Tom,

Thank you for reaching out.

Could you please tell us the following:

- Which OS version are you using?

- Which OpenVINO™ toolkit version are you using?

Regards,

David

[Deblauwe__Tom]

Hello,

I am using openvino 2019_R3.1 from https://github.com/opencv/dldt. I compiled it on my own raspberry pi OS made with a buildroot-like tool called OpenADK, but that is not very relevant for my question as far as I see.

Best regards,

Tom,

[Sahira_Intel]

Hi Tom,

The NCS essentially loads the model into memory to be able to run inference on that model. Keeping the model information in memory helps when running a series of inferences and when multiple models are loaded. It also helps with inference speeds.

The way OpenVINO is designed, memory is also allocated for things like fifo queues for input and output data. There is also some amount of memory set aside for intermediate layer processing. 

If your model is only 40 MB, you should have more than enough memory available since the NCS1 has 500 MB of internal memory. 

I hope this information is helpful.

Best Regards,

Sahira