Community
cancel
Showing results for 
Search instead for 
Did you mean: 
Highlighted
Beginner
298 Views

Failed to find booted device after boot

Hi,

    I'm running dockers deployed using azure as in https://github.com/Azure-Samples/onnxruntime-iot-edge/blob/master/README-ONNXRUNTIME-OpenVINO.md#pha...

Container starts successfully and is able to download YOLO onto the device. Then reboots the device but never finds it back within the container.

I have started the docker in privileged mode. I also found that every time the device reboots, it comes back with the same product ID: Bus 002 Device 047: ID 03e7:2485 . Still i tried keeping the product ID blank in the udev rules hoping my application accepts all product ID with  03e7 vendor ID, but that didn't help. I tried changing the mode to 0666, that didn't help either. 

Is there anything i miss ? Below are logs from the container application:

 

 

2020-06-29 21:01:48.414589245 [W:onnxruntime:, graph.cc:814 Graph] Initializer convolution8_W appears in graph inputs and will not be treated as constant value/weight. This may fail some of the graph optimizations, like const folding. Move it out of graph inputs if there is no need to override it, by either re-generating the model with latest exporter/converter or with the tool onnxruntime/tools/python/remove_initializer_from_input.py.
2020-06-29 21:01:48.414607717 [W:onnxruntime:, graph.cc:814 Graph] Initializer convolution8_B appears in graph inputs and will not be treated as constant value/weight. This may fail some of the graph optimizations, like const folding. Move it out of graph inputs if there is no need to override it, by either re-generating the model with latest exporter/converter or with the tool onnxruntime/tools/python/remove_initializer_from_input.py.
loaded after 1.7521090507507324 s
CLOUD STORAGE STATUS: True
trying to make IOT Hub manager
INITIALIZED AFTER 1.0028696060180664 s
* Serving Flask app "main" (lazy loading)
* Environment: production
WARNING: This is a development server. Do not use it in a production deployment.
Use a production WSGI server instead.
* Debug mode: off
* Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
[WARN] 2020-06-29T21:01:51z src/ngraph/frontend/onnx_import/ops_bridge.cpp 190 Domain 'ai.onnx.preview.training' not recognized by nGraph
[WARN] 2020-06-29T21:01:51z src/ngraph/frontend/onnx_import/ops_bridge.cpp 190 Domain 'com.microsoft.nchwc' not recognized by nGraph
[WARN] 2020-06-29T21:01:51z src/ngraph/frontend/onnx_import/ops_bridge.cpp 190 Domain 'com.microsoft' not recognized by nGraph
[WARN] 2020-06-29T21:01:51z src/ngraph/frontend/onnx_import/ops_bridge.cpp 190 Domain 'ai.onnx.ml' not recognized by nGraph
[WARN] 2020-06-29T21:01:51z src/ngraph/frontend/onnx_import/ops_bridge.cpp 190 Domain 'com.microsoft.mlfeaturizers' not recognized by nGraph
[WARN] 2020-06-29T21:01:51z src/ngraph/frontend/onnx_import/ops_bridge.cpp 190 Domain 'ai.onnx.training' not recognized by nGraph
E: [ncAPI] [ 528983] [python] ncDeviceOpen:1003 Failed to find booted device after boot
2020-06-29 21:02:08.991818030 [E:onnxruntime:, sequential_executor.cc:281 Execute] Non-zero status code returned while running OpenVINO-EP-subgraph_1 node. Name:'OpenVINOExecutionProvider_OpenVINO-EP-subgraph_1_0' Status Message: /code/onnxruntime/onnxruntime/core/providers/openvino/backends/basic_backend.cc:41 onnxruntime::openvino_ep::BasicBackend::BasicBackend(const onnx::ModelProto&, onnxruntime::openvino_ep::GlobalContext&, const onnxruntime::openvino_ep::SubGraphContext&) [OpenVINO-EP] Exception while Loading Network for graph: OpenVINOExecutionProvider_OpenVINO-EP-subgraph_1_0Can not init Myriad device: NC_ERROR

 

Thanks

0 Kudos
4 Replies
Highlighted
Beginner
278 Views

I solved the problem of not finding the booted device. Mapped the container's dev with host dev.

Now, the errors are as below. Can somebody help please ...

 

2020-06-30 18:52:10.059441680 [W:onnxruntime:, graph.cc:814 Graph] Initializer convolution8_W appears in graph inputs and will not be treated as constant value/weight. This may fail some of the graph optimizations, like const folding. Move it out of graph inputs if there is no need to override it, by either re-generating the model with latest exporter/converter or with the tool onnxruntime/tools/python/remove_initializer_from_input.py.
2020-06-30 18:52:10.059456585 [W:onnxruntime:, graph.cc:814 Graph] Initializer convolution8_B appears in graph inputs and will not be treated as constant value/weight. This may fail some of the graph optimizations, like const folding. Move it out of graph inputs if there is no need to override it, by either re-generating the model with latest exporter/converter or with the tool onnxruntime/tools/python/remove_initializer_from_input.py.
loaded after 1.5334908962249756 s
CLOUD STORAGE STATUS: True
trying to make IOT Hub manager
INITIALIZED AFTER 1.0015590190887451 s
* Serving Flask app "main" (lazy loading)
* Environment: production
WARNING: This is a development server. Do not use it in a production deployment.
Use a production WSGI server instead.
* Debug mode: off
* Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
[WARN] 2020-06-30T18:52:13z src/ngraph/frontend/onnx_import/ops_bridge.cpp 190 Domain 'ai.onnx.preview.training' not recognized by nGraph
[WARN] 2020-06-30T18:52:13z src/ngraph/frontend/onnx_import/ops_bridge.cpp 190 Domain 'com.microsoft.nchwc' not recognized by nGraph
[WARN] 2020-06-30T18:52:13z src/ngraph/frontend/onnx_import/ops_bridge.cpp 190 Domain 'com.microsoft' not recognized by nGraph
[WARN] 2020-06-30T18:52:13z src/ngraph/frontend/onnx_import/ops_bridge.cpp 190 Domain 'ai.onnx.ml' not recognized by nGraph
[WARN] 2020-06-30T18:52:13z src/ngraph/frontend/onnx_import/ops_bridge.cpp 190 Domain 'com.microsoft.mlfeaturizers' not recognized by nGraph
[WARN] 2020-06-30T18:52:13z src/ngraph/frontend/onnx_import/ops_bridge.cpp 190 Domain 'ai.onnx.training' not recognized by nGraph
2020-06-30 18:52:18.611212446 [E:onnxruntime:, sequential_executor.cc:281 Execute] Non-zero status code returned while running OpenVINO-EP-subgraph_1 node. Name:'OpenVINOExecutionProvider_OpenVINO-EP-subgraph_1_0' Status Message: /code/onnxruntime/onnxruntime/core/providers/openvino/backends/basic_backend.cc:41 onnxruntime::openvino_ep::BasicBackend::BasicBackend(const onnx::ModelProto&, onnxruntime::openvino_ep::GlobalContext&, const onnxruntime::openvino_ep::SubGraphContext&) [OpenVINO-EP] Exception while Loading Network for graph: OpenVINOExecutionProvider_OpenVINO-EP-subgraph_1_0Can not init Myriad device: NC_ERROR

EXCEPTION: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Non-zero status code returned while running OpenVINO-EP-subgraph_1 node. Name:'OpenVINOExecutionProvider_OpenVINO-EP-subgraph_1_0' Status Message: /code/onnxruntime/onnxruntime/core/providers/openvino/backends/basic_backend.cc:41 onnxruntime::openvino_ep::BasicBackend::BasicBackend(const onnx::ModelProto&, onnxruntime::openvino_ep::GlobalContext&, const onnxruntime::openvino_ep::SubGraphContext&) [OpenVINO-EP] Exception while Loading Network for graph: OpenVINOExecutionProvider_OpenVINO-EP-subgraph_1_0Can not init Myriad device: NC_ERROR

 

0 Kudos
Highlighted
Beginner
269 Views

This is strange, i removed the dev map and for some reason this print "Failed to find booted device after boot" doesn't always appear. 

Can somebody tell me if the below logs appear after the model is loaded and rebooted or did the model not load at all ?

The device is really hot, so it surely has got loaded something and executing.

cognized by nGraph
2020-06-30 20:19:34.157581736 [E:onnxruntime:, sequential_executor.cc:281 Execute] Non-zero status code returned while running OpenVINO-EP-subgraph_1 node. Name:'OpenVINOExecutionProvider_OpenVINO-EP-subgraph_1_0' Status Message: /code/onnxruntime/onnxruntime/core/providers/openvino/backends/basic_backend.cc:41 onnxruntime::openvino_ep::BasicBackend::BasicBackend(const onnx::ModelProto&, onnxruntime::openvino_ep::GlobalContext&, const onnxruntime::openvino_ep::SubGraphContext&) [OpenVINO-EP] Exception while Loading Network for graph: OpenVINOExecutionProvider_OpenVINO-EP-subgraph_1_0Can not init Myriad device: NC_ERROR

0 Kudos
Highlighted
Beginner
241 Views

Ok, i'm able to reproduce this consistently. The python print disappears if i map container's dev and host dev. However with the following error

" Exception while Loading Network for graph: OpenVINOExecutionProvider_OpenVINO-EP-subgraph_1_0Can not init Myriad device: NC_ERROR" 

Thanks

Srinivas

0 Kudos
Highlighted
Moderator
214 Views

Greetings,


It seems that your problem occurs due to some USB settings/configuration for the stick.

The same thing often occurs with virtual machine which could be solved by correctly configure the usb to appropriate settings.


You may refer here for dockers configuration:

https://docs.sevenbridges.com/docs/mount-a-usb-drive-in-a-docker-container


Sincerely,

Iffa


0 Kudos