Failed to find booted device after boot

[SrinivasBakki]

Hi,

    I'm running dockers deployed using azure as in https://github.com/Azure-Samples/onnxruntime-iot-edge/blob/master/README-ONNXRUNTIME-OpenVINO.md#phase-two-model-deployment-with-onxn-runtime--openvino

Container starts successfully and is able to download YOLO onto the device. Then reboots the device but never finds it back within the container.

I have started the docker in privileged mode. I also found that every time the device reboots, it comes back with the same product ID: Bus 002 Device 047: ID 03e7:2485 . Still i tried keeping the product ID blank in the udev rules hoping my application accepts all product ID with  03e7 vendor ID, but that didn't help. I tried changing the mode to 0666, that didn't help either. 

Is there anything i miss ? Below are logs from the container application:

2020-06-29 21:01:48.414589245 [W:onnxruntime:, graph.cc:814 Graph] Initializer convolution8_W appears in graph inputs and will not be treated as constant value/weight. This may fail some of the graph optimizations, like const folding. Move it out of graph inputs if there is no need to override it, by either re-generating the model with latest exporter/converter or with the tool onnxruntime/tools/python/remove_initializer_from_input.py.
2020-06-29 21:01:48.414607717 [W:onnxruntime:, graph.cc:814 Graph] Initializer convolution8_B appears in graph inputs and will not be treated as constant value/weight. This may fail some of the graph optimizations, like const folding. Move it out of graph inputs if there is no need to override it, by either re-generating the model with latest exporter/converter or with the tool onnxruntime/tools/python/remove_initializer_from_input.py.
loaded after 1.7521090507507324 s
CLOUD STORAGE STATUS: True
trying to make IOT Hub manager
INITIALIZED AFTER 1.0028696060180664 s
* Serving Flask app "main" (lazy loading)
* Environment: production
WARNING: This is a development server. Do not use it in a production deployment.
Use a production WSGI server instead.
* Debug mode: off
* Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
[WARN] 2020-06-29T21:01:51z src/ngraph/frontend/onnx_import/ops_bridge.cpp 190 Domain 'ai.onnx.preview.training' not recognized by nGraph
[WARN] 2020-06-29T21:01:51z src/ngraph/frontend/onnx_import/ops_bridge.cpp 190 Domain 'com.microsoft.nchwc' not recognized by nGraph
[WARN] 2020-06-29T21:01:51z src/ngraph/frontend/onnx_import/ops_bridge.cpp 190 Domain 'com.microsoft' not recognized by nGraph
[WARN] 2020-06-29T21:01:51z src/ngraph/frontend/onnx_import/ops_bridge.cpp 190 Domain 'ai.onnx.ml' not recognized by nGraph
[WARN] 2020-06-29T21:01:51z src/ngraph/frontend/onnx_import/ops_bridge.cpp 190 Domain 'com.microsoft.mlfeaturizers' not recognized by nGraph
[WARN] 2020-06-29T21:01:51z src/ngraph/frontend/onnx_import/ops_bridge.cpp 190 Domain 'ai.onnx.training' not recognized by nGraph
E: [ncAPI] [ 528983] [python] ncDeviceOpen:1003 Failed to find booted device after boot
2020-06-29 21:02:08.991818030 [E:onnxruntime:, sequential_executor.cc:281 Execute] Non-zero status code returned while running OpenVINO-EP-subgraph_1 node. Name:'OpenVINOExecutionProvider_OpenVINO-EP-subgraph_1_0' Status Message: /code/onnxruntime/onnxruntime/core/providers/openvino/backends/basic_backend.cc:41 onnxruntime::openvino_ep::BasicBackend::BasicBackend(const onnx::ModelProto&, onnxruntime::openvino_ep::GlobalContext&, const onnxruntime::openvino_ep::SubGraphContext&) [OpenVINO-EP] Exception while Loading Network for graph: OpenVINOExecutionProvider_OpenVINO-EP-subgraph_1_0Can not init Myriad device: NC_ERROR

Thanks

[SrinivasBakki]

I solved the problem of not finding the booted device. Mapped the container's dev with host dev.

Now, the errors are as below. Can somebody help please ...

2020-06-30 18:52:10.059441680 [W:onnxruntime:, graph.cc:814 Graph] Initializer convolution8_W appears in graph inputs and will not be treated as constant value/weight. This may fail some of the graph optimizations, like const folding. Move it out of graph inputs if there is no need to override it, by either re-generating the model with latest exporter/converter or with the tool onnxruntime/tools/python/remove_initializer_from_input.py.
2020-06-30 18:52:10.059456585 [W:onnxruntime:, graph.cc:814 Graph] Initializer convolution8_B appears in graph inputs and will not be treated as constant value/weight. This may fail some of the graph optimizations, like const folding. Move it out of graph inputs if there is no need to override it, by either re-generating the model with latest exporter/converter or with the tool onnxruntime/tools/python/remove_initializer_from_input.py.
loaded after 1.5334908962249756 s
CLOUD STORAGE STATUS: True
trying to make IOT Hub manager
INITIALIZED AFTER 1.0015590190887451 s
* Serving Flask app "main" (lazy loading)
* Environment: production
WARNING: This is a development server. Do not use it in a production deployment.
Use a production WSGI server instead.
* Debug mode: off
* Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
[WARN] 2020-06-30T18:52:13z src/ngraph/frontend/onnx_import/ops_bridge.cpp 190 Domain 'ai.onnx.preview.training' not recognized by nGraph
[WARN] 2020-06-30T18:52:13z src/ngraph/frontend/onnx_import/ops_bridge.cpp 190 Domain 'com.microsoft.nchwc' not recognized by nGraph
[WARN] 2020-06-30T18:52:13z src/ngraph/frontend/onnx_import/ops_bridge.cpp 190 Domain 'com.microsoft' not recognized by nGraph
[WARN] 2020-06-30T18:52:13z src/ngraph/frontend/onnx_import/ops_bridge.cpp 190 Domain 'ai.onnx.ml' not recognized by nGraph
[WARN] 2020-06-30T18:52:13z src/ngraph/frontend/onnx_import/ops_bridge.cpp 190 Domain 'com.microsoft.mlfeaturizers' not recognized by nGraph
[WARN] 2020-06-30T18:52:13z src/ngraph/frontend/onnx_import/ops_bridge.cpp 190 Domain 'ai.onnx.training' not recognized by nGraph
2020-06-30 18:52:18.611212446 [E:onnxruntime:, sequential_executor.cc:281 Execute] Non-zero status code returned while running OpenVINO-EP-subgraph_1 node. Name:'OpenVINOExecutionProvider_OpenVINO-EP-subgraph_1_0' Status Message: /code/onnxruntime/onnxruntime/core/providers/openvino/backends/basic_backend.cc:41 onnxruntime::openvino_ep::BasicBackend::BasicBackend(const onnx::ModelProto&, onnxruntime::openvino_ep::GlobalContext&, const onnxruntime::openvino_ep::SubGraphContext&) [OpenVINO-EP] Exception while Loading Network for graph: OpenVINOExecutionProvider_OpenVINO-EP-subgraph_1_0Can not init Myriad device: NC_ERROR

EXCEPTION: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Non-zero status code returned while running OpenVINO-EP-subgraph_1 node. Name:'OpenVINOExecutionProvider_OpenVINO-EP-subgraph_1_0' Status Message: /code/onnxruntime/onnxruntime/core/providers/openvino/backends/basic_backend.cc:41 onnxruntime::openvino_ep::BasicBackend::BasicBackend(const onnx::ModelProto&, onnxruntime::openvino_ep::GlobalContext&, const onnxruntime::openvino_ep::SubGraphContext&) [OpenVINO-EP] Exception while Loading Network for graph: OpenVINOExecutionProvider_OpenVINO-EP-subgraph_1_0Can not init Myriad device: NC_ERROR

[SrinivasBakki]

This is strange, i removed the dev map and for some reason this print "Failed to find booted device after boot" doesn't always appear. 

Can somebody tell me if the below logs appear after the model is loaded and rebooted or did the model not load at all ?

The device is really hot, so it surely has got loaded something and executing.

cognized by nGraph
2020-06-30 20:19:34.157581736 [E:onnxruntime:, sequential_executor.cc:281 Execute] Non-zero status code returned while running OpenVINO-EP-subgraph_1 node. Name:'OpenVINOExecutionProvider_OpenVINO-EP-subgraph_1_0' Status Message: /code/onnxruntime/onnxruntime/core/providers/openvino/backends/basic_backend.cc:41 onnxruntime::openvino_ep::BasicBackend::BasicBackend(const onnx::ModelProto&, onnxruntime::openvino_ep::GlobalContext&, const onnxruntime::openvino_ep::SubGraphContext&) [OpenVINO-EP] Exception while Loading Network for graph: OpenVINOExecutionProvider_OpenVINO-EP-subgraph_1_0Can not init Myriad device: NC_ERROR

[SrinivasBakki]

Ok, i'm able to reproduce this consistently. The python print disappears if i map container's dev and host dev. However with the following error

" Exception while Loading Network for graph: OpenVINOExecutionProvider_OpenVINO-EP-subgraph_1_0Can not init Myriad device: NC_ERROR" 

Thanks

Srinivas

[OyamingO]

Did you solve that problem now?

[Iffa_Intel]

Greetings,

It seems that your problem occurs due to some USB settings/configuration for the stick.

The same thing often occurs with virtual machine which could be solved by correctly configure the usb to appropriate settings.

You may refer here for dockers configuration:

https://docs.sevenbridges.com/docs/mount-a-usb-drive-in-a-docker-container

Sincerely,

Iffa