Please provide a minimal

wsla · ‎01-29-2019

Hi,

We have several units deployed in production running Windows 10 IoT Enterprise with Movidius Compute Stick. The application is a Windows service written in C# and we are using OpenVino R5.

Sometimes right after boot we get error "Failed to allocate graph: NC_OUT_OF_MEMORY" from:

executableNetwork = plugin.LoadNetwork(network, {});

Most of the times it works though (90% or so). The application keeps trying to load the model and sometimes it succeeds after few attempts - but not always. According to the description on Movidius website this error means that "the host is out of memory". I don't quite understand how is it possible that Movidius is out of memory right after boot. What could I do to either prevent this error or to recover from it?

We are using Resnet50 converted from Tensorflow.

Thanks in advance

Dmitry_K_Intel3 · ‎01-29-2019

Please provide a minimal working example so we can reproduce it.

It it possible to reproduce it only with a single model ResNet-50? Is there a source to download it?

Which Stick version is used? Is it NCS or NCS2?

Shubha_R_Intel · ‎01-30-2019

There are basically two cases for this error

Some malloc call on the host returned zero pointer. I assume this is not the case
On the Myriad side there is no continuous block of free memory that is big enough to store the model. If user’s application loads/unloads graphs multiple times then it might be result of device memory fragmentation.

wsla · ‎01-31-2019

Hi Dmitry and Shubha,

Thank you for your replies.

Dmitry , I will extract a minimal working example from our application and get back to you with it and converted model. We are using NCS 1.

Shubha, in regards to point 2 how is it possible that there is no memory right after start? In case of this exception what would be the best way to recover- is it possible to free all allocations in NCS? And can I somehow access info about current memory usage in the NCS - I could then trace it out.

Thank you

Dmitry_K_Intel3 · ‎01-31-2019

There is a thread where we figured out that in case of multithreading initialization of Myriad plugin we can face the same error: https://github.com/opencv/opencv/issues/13715.

Perhaps, your application has similar initialization and you can solve it by loading a plugin in the main thread.

wsla · ‎01-31-2019

Hi Dmitry,

We only have a single thread in C# application that is accessing C++ code for inference operations. I have attached C++ structure of our code (minus any exception handling or tracing). It is based on the classification_sample.

If we encounter any exception in the constructor, we delete the whole object and start from scratch. In case of this error exception is thrown in line 33: "executableNetwork = plugin.LoadNetwork(network, {});"

if we want to recover from this error should we start from scratch (reload plugin, re-read network..) or should we keep trying to call "plugin.LoadNetwork(network, {})"?

If there is any way to trace the current state of the NCS (memory usage etc) than I could perhaps get back to you with more useful information?

Thank you

wsla · ‎02-05-2019

Hi Dmitry,

Did you have a chance to check the sample I attached to my last comment?

I still don't know if there is any tracing inside OpenVino that I could enable to get more info about the problem?

I also noticed (in the logs from production) that sometimes if application doesn't recover from that issue (after few attempts) it tends to get stucked (on the same line which returns NC_OUT_OF_MEMORY) for exactly 1 hour, then returns error and never recovers from it, killing of this thread doesnt help either.

Dmitry_K_Intel3 · ‎02-05-2019

wsla, What about the model? Is it ResNet-50 from Model Zoo? https://github.com/opencv/open_model_zoo/blob/2018/model_downloader/list_topologies.yml

And there is no main in your *.cpp file. Can you please make it more reproducible.

wsla · ‎03-12-2019

Hi Dmitry,

I have finally identified what is causing this issue - usb extension cable:

- stick plugged directly into the machine - no errors

- stick plugged in via a low-quality usb 2.0 extension cable (1m or shorter) - we get NC_OUT_OF_MEMORY maybe 1 in 100 times on application start

- stick plugged in via a low-quality usb 2.0 extension cable (2m) - application can't find the device (NC_DEVICE_NOT_FOUND)

- stick plugged in via a reasonable quality 2m usb 3.0 cable (https://www.amazon.com/Syncwire-Extension-Extender-Transfer-Playstation/dp/B072M721QD) - we get NC_OUT_OF_MEMORY on approximately 30 - 50% of application starts. When plugging the stick via 4m cable (2 * 2m extension) we get it on roughly 75% of application starts.

Considering that after NC_OUT_OF_MEMORY error (one or multiple - as application keeps trying to load the model) when using 2m or 4m usb 3.0 cable application can usually finally load the model and run predictions normally for a long period of time (several hours) I have a feeling that there must be some internal time-out when loading model in OpenVINO that is just triggered because of the additional usb latency?

Thanks

wsla · ‎03-13-2019

Here is some tracing from when trying to load model to Movidius connected to the on-board computer (Intel Atom) via 1m certified USB 3.0 cable:

[2019-03-14 11:01:47.728736] [0x00000fd8] [info]    Trying to load inference engine.
[2019-03-14 11:01:47.754773] [0x00000fd8] [info]    Inference engine plugin loaded.
[2019-03-14 11:01:47.754773] [0x00000fd8] [debug]   Reading network graph.
[2019-03-14 11:01:47.777810] [0x00000fd8] [debug]   Reading network weights.
[2019-03-14 11:01:48.202932] [0x00000fd8] [debug]   Loading network.                           
[2019-03-14 11:01:48.220961] [0x00000fd8] [info]    Intermediate representation loaded.
[2019-03-14 11:01:48.238486] [0x00000fd8] [debug]   Configuring input info.
[2019-03-14 11:01:48.239485] [0x00000fd8] [debug]   Preparing output blobs.
[2019-03-14 11:01:48.263521] [0x00000fd8] [debug]   Setting precision.
[2019-03-14 11:01:48.264522] [0x00000fd8] [info]    Input and output loaded.
[1;32mOpening device #0[0m
[35mE: [watchdog] [         0] sendPingMessage:164     Failed send ping message: X_LINK_ERROR[35mE: [ncAPI] [         0] getDevAttributes:408        Failed to write data, rc: X_LINK_COMMUNICATION_FAIL[0m
[0m                                                                                           [1;32mDevice #0 MYRIAD-2 allocated[0m
[1;33m[VPU] GraphTransformer : transform network Norilsk_20190121_01 at ..\src\vpu\graph_transformer\src\frontend\parse_network.cpp:124[0m
[1;33m[VPU] GraphTransformer : convert input resnet50_input to FP16 at ..\src\vpu\graph_transformer\src\frontend\in_out_convert.cpp:126[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/conv1/convolution at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/activation_1/Relu at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/max_pooling2d_1/MaxPool at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/res2a_branch2a/convolution at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/activation_2/Relu at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/res2a_branch2b/convolution at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/activation_3/Relu at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/res2a_branch2c/convolution at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/res2a_branch1/convolution at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/add_1/add at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/activation_4/Relu at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/res2b_branch2a/convolution at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/activation_5/Relu at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/res2b_branch2b/convolution at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/activation_6/Relu at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/res2b_branch2c/convolution at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/add_2/add at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/activation_7/Relu at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/res2c_branch2a/convolution at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/activation_8/Relu at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/res2c_branch2b/convolution at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/activation_9/Relu at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/res2c_branch2c/convolution at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/add_3/add at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/activation_10/Relu at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/res3a_branch2a/convolution at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/activation_11/Relu at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/res3a_branch2b/convolution at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/activation_12/Relu at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/res3a_branch2c/convolution at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/res3a_branch1/convolution at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/add_4/add at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/activation_13/Relu at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/res3b_branch2a/convolution at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/activation_14/Relu at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/res3b_branch2b/convolution at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/activation_15/Relu at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/res3b_branch2c/convolution at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/add_5/add at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/activation_16/Relu at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/res3c_branch2a/convolution at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/activation_17/Relu at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/res3c_branch2b/convolution at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/activation_18/Relu at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/res3c_branch2c/convolution at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/add_6/add at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/activation_19/Relu at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/res3d_branch2a/convolution at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/activation_20/Relu at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/res3d_branch2b/convolution at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/activation_21/Relu at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/res3d_branch2c/convolution at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/add_7/add at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/activation_22/Relu at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/res4a_branch2a/convolution at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/activation_23/Relu at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/res4a_branch2b/convolution at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/activation_24/Relu at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/res4a_branch2c/convolution at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/res4a_branch1/convolution at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/add_8/add at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/activation_25/Relu at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/res4b_branch2a/convolution at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/activation_26/Relu at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/res4b_branch2b/convolution at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/activation_27/Relu at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/res4b_branch2c/convolution at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/add_9/add at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/activation_28/Relu at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/res4c_branch2a/convolution at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/activation_29/Relu at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/res4c_branch2b/convolution at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/activation_30/Relu at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/res4c_branch2c/convolution at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/add_10/add at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/activation_31/Relu at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/res4d_branch2a/convolution at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/activation_32/Relu at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/res4d_branch2b/convolution at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/activation_33/Relu at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/res4d_branch2c/convolution at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/add_11/add at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/activation_34/Relu at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/res4e_branch2a/convolution at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/activation_35/Relu at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/res4e_branch2b/convolution at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/activation_36/Relu at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/res4e_branch2c/convolution at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/add_12/add at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/activation_37/Relu at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/res4f_branch2a/convolution at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/activation_38/Relu at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/res4f_branch2b/convolution at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/activation_39/Relu at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/res4f_branch2c/convolution at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/add_13/add at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/activation_40/Relu at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/res5a_branch2a/convolution at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/activation_41/Relu at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/res5a_branch2b/convolution at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/activation_42/Relu at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/res5a_branch2c/convolution at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/res5a_branch1/convolution at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/add_14/add at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/activation_43/Relu at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/res5b_branch2a/convolution at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/activation_44/Relu at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/res5b_branch2b/convolution at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/activation_45/Relu at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/res5b_branch2c/convolution at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/add_15/add at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/activation_46/Relu at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/res5c_branch2a/convolution at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/activation_47/Relu at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/res5c_branch2b/convolution at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/activation_48/Relu at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/res5c_branch2c/convolution at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/add_16/add at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer resnet50/activation_49/Relu at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer average_pooling2d_1/AvgPool at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer flatten_1/Reshape at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer dense_1/MatMul at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : try to parse layer dense_1/Softmax at ..\src\vpu\graph_transformer\src\graph_transformer_impl.cpp:541[0m
[1;33m[VPU] GraphTransformer : convert output dense_1/Softmax from FP16 at ..\src\vpu\graph_transformer\src\frontend\in_out_convert.cpp:196[0m
WinUsb_ReadPipe: System err 31
[35mE: [xLink] [         0] dispatcherEventReceive:308 dispatcherEventReceive() Read failed -2 | event 0000003B35DFF850 USB_WRITE_RESP
[0m
[35mE: [xLink] [         0] eventReader:256    eventReader stopped[0m
[35mE: [watchdog] [         0] sendPingMessage:164     Failed send ping message: X_LINK_ERROR[0m                                      
[35mE: [watchdog] [         0] sendPingMessage:164     Failed send ping message: X_LINK_ERROR[0m
[1;32m[VPU] GraphTransformer : blobSize=47173508[0m
[35mE: [ncAPI] [         0] ncGraphAllocate:1348       The graph file is bigger than the device memory[0m
[2019-03-14 11:01:53.773124] [0x00000fd8] [error]   Inference engine exception while trying to load model into the plugin (D:\model\Norilsk_20190121_01\Norilsk_20190121_01) (exception: Failed to allocate graph: NC_OUT_OF_MEMORY)                                                         [2019-03-14 11:01:53.866263] [0x0000028c] [debug]   Nulling infer request.                     
[2019-03-14 11:01:53.895305] [0x0000028c] [debug]   Nulling executable network.
[2019-03-14 11:01:53.908322] [0x0000028c] [debug]   Nulling plugin.
[2019-03-14 11:01:53.985434] [0x0000028c] [debug]   Inference destructor finished.
--------------------------------------------------
Model loading statistics:
Attempts: 1
Model loaded successfully: 0
NC_OUT_OF_MEMORY: 1
Other errors: 0
--------------------------------------------------

Shubha_R_Intel · ‎03-28-2019

Dear wsla:

Perhaps this forum post will also help you :

https://software.intel.com/en-us/forums/computer-vision/topic/805070

Movidius - NC_OUT_OF_MEMORY