RPi4 requires reboot to load the model on the plugin for the second try

[Sharma__Robel]

Environment:

  OpenVino: Master from the github.com.

  OpenCV: 4.1

  NCS: Version 2.

   Host: RPi4

Scenario: Used the object detection async sample to run a model. Worked fine for the first execution.

Kill the process/binary and start the process again. Try this for several of the time.

Then the following model load failure will be seen -

Dec 04 05:17:18 raspberrypi streamlauncher[8595]: InferenceEngine:
Dec 04 05:17:18 raspberrypi streamlauncher[8595]:         API version ............ 2.1
Dec 04 05:17:18 raspberrypi streamlauncher[8595]:         Build .................. custom_2019_733dae46cc2271da4a743b87c0d3facf328b9c10
Dec 04 05:17:18 raspberrypi streamlauncher[8595]:         Description ....... API
Dec 04 05:17:18 raspberrypi streamlauncher[8595]: [ INFO ] Loading plugin
Dec 04 05:17:18 raspberrypi streamlauncher[8595]: [ INFO ] Plugin load done
Dec 04 05:17:18 raspberrypi streamlauncher[8595]: [ INFO ] Loading network files: ./face-detection-adas-0001.xml
Dec 04 05:17:18 raspberrypi streamlauncher[8595]: [ INFO ] Batch size is forced to  1.
Dec 04 05:17:18 raspberrypi streamlauncher[8595]: [ INFO ] Checking that the inputs are as the demo expects
Dec 04 05:17:18 raspberrypi streamlauncher[8595]: [ INFO ] Checking that the outputs are as the demo expects
Dec 04 05:17:18 raspberrypi streamlauncher[8595]: Reading labels size: 0 <==> classes: 2
Dec 04 05:17:18 raspberrypi streamlauncher[8595]: [ INFO ] Loading model to the plugin
Dec 04 05:17:20 raspberrypi streamlauncher[8595]: [ INFO ] Loading model failed

dmesg output:

[ 8814.121648] usb 1-1.1: USB disconnect, device number 24
[ 8816.866553] usb 1-1.1: new high-speed USB device number 25 using xhci_hcd
[ 8816.997205] usb 1-1.1: New USB device found, idVendor=03e7, idProduct=2485, bcdDevice= 0.01
[ 8816.997220] usb 1-1.1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[ 8816.997233] usb 1-1.1: Product: Movidius MyriadX
[ 8816.997245] usb 1-1.1: Manufacturer: Movidius Ltd.

[JesusE_Intel]

Hi Robel,

Thank you for reaching out. I see the Intel® Neural Compute Stick 2 (Intel® NCS 2) was disconnected during the execution of the sample and reconnected after the running process. It might be due to a lack of power from the RPi4. Can you try to run the model again using the Intel® NCS 2 with a powered 3.0 USB Hub.

Regards,

Jesus

[Sharma__Robel]

Hi Jesus,

Thanks for your prompt response. I just did a test using a script available in the gist

root@AnyConnectAICamera:/opt/anyconnect/ac-stream/RPi/Launcher# ./throttled.sh
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
Status: 0x80000
Undervolted:
   Now: NO
   Run: NO
Throttled:
   Now: NO
   Run: NO
Frequency Capped:
   Now: NO
   Run: NO

Seems like the voltage is OK.

FYI, we are a company that currently trying to deploy the inference in the edge and doing a developers test with around 40 units now. https://anyconnect.com/computer-vision/

We appreciate your help.

Thanks,

- Robel

[Sharma__Robel]

Hi Jesus,

Today I am also seeing the following issues,

Dec 09 05:00:16 raspberrypi streamlauncher[4537]: E: [ncAPI] [    592641] [StreamApp] ncGraphQueueInference:3973        Can't send trigger request
Dec 09 05:00:16 raspberrypi streamlauncher[4537]: E: [xLink] [    592642] [EventRead00Thr] eventReader:218        eventReader thread stopped (err -4)
Dec 09 05:00:16 raspberrypi streamlauncher[4537]: E: [global] [    592642] [Scheduler00Thr] dispatcherEventSend:1013        Write failed -4
Dec 09 05:00:16 raspberrypi streamlauncher[4537]:
Dec 09 05:00:16 raspberrypi streamlauncher[4537]: E: [xLink] [    592642] [Scheduler00Thr] eventSchedulerRun:576        Dispatcher received NULL event!
Dec 09 05:00:16 raspberrypi streamlauncher[4537]: E: [global] [    592642] [StreamApp] XLinkReadDataWithTimeOut:1494        Event data is invalid
Dec 09 05:00:16 raspberrypi streamlauncher[4537]: E: [ncAPI] [    592642] [StreamApp] ncFifoReadElem:3510        Packet reading is failed.
Dec 09 05:00:16 raspberrypi streamlauncher[4537]: E: [watchdog] [    592642] [WatchdogThread] sendPingMessage:121        Failed send ping message: X_LINK_TIMEOUT
Dec 09 05:00:16 raspberrypi streamlauncher[4537]: E: [watchdog] [    592643] [WatchdogThread] sendPingMessage:121        Failed send ping message: X_LINK_ERROR
Dec 09 05:00:16 raspberrypi streamlauncher[4537]: E: [watchdog] [    592643] [WatchdogThread] watchdog_routine:315        [0x1660778] device, not respond, removing from watchdog

- Robel

[JesusE_Intel]

Hi Robel,

Thank you for your response. Could you please answer some question we have for you:

• Are the error messages shown as soon as the code starts or after a certain period of time?
• Do you see these errors using one Intel® Neural Compute Stick 2 or all the units?
• Are you using any USB hubs or connecting the Intel® Neural Compute Stick 2 directly to the Pi4?
• Do you see the same behavior with our OpenVINO Toolkit for Rasbian OS package?

Regards,

Jesus

[Sharma__Robel]

Are the error messages shown as soon as the code starts or after a certain period of time

- After a certain period of time.

Do you see these errors using one Intel® Neural Compute Stick 2 or all the units?

- On the development environment, we tried 3 different stick in 3 different Pi 4 and the result is the same.

Are you using any USB hubs or connecting the Intel® Neural Compute Stick 2 directly to the Pi4?

- Direct RPi 4 USB 3.0 port.

Do you see the same behaviour with our OpenVINO Toolkit for Rasbian OS package?

- Do not know. Around 1 month ago when I tried the released OpenVINO Toolkit it failed on my RPi 4. So I have to build it from the source.

- Robel

[JesusE_Intel]

Hi Robel,

I tested the object_detection_sample_ssd on a Raspberry Pi 4 with the OpenVINO Toolkit for Raspbian OS and I was not able to replicate your issue after executing the demo multiple times. Could you try using the pre-built package and see if you see the same behavior? 

Kill the process/binary and start the process again. Try this for several of the time.

Why do you kill the process after it starts? The demo will end once it loads the model, runs inference, and saves the output file. When the model is loaded to the device, it will reboot and connect as a VSP loopback device. Once the demo completes, the device will reconnect as Myriad X as seen in your dmesg.

Regards,

Jesus

[Sharma__Robel]

Hi Jesus,

Thanks for your fast response.

Good news first!

- I tried with latest dldt OpenVINO from the intel repo and seems the occurrence is around 1 out of 10.

Now the bad news.

- Our use case is a kind of general Car dashcam scenario and wearable camera. So the camera start on in ignition and goes down when the car is stopped. Other than that, I had to kill the process for the test or reproducing the issue. But it happens on the long run too as I mentioned in one of my comment.

Regards,

Robel Sharma

[JesusE_Intel]

Hi Robel,

What you are seeing with the device not recovering after not closing properly may be related to a bug we are investigating. Let me find out if there is a current workaround. Regarding the xlink error, how long does the program run until the error is seen? Which network are you using? I want to try to reproduce from my end. 

Regards,

Jesus

[Sharma__Robel]

Let me find out if there is a current workaround.

Thank you, Jesus.

Regarding the xlink error, how long does the program run until the error is seen?

It takes around 4 to 5 hours of inferencing on live streaming.

Which network are you using?

face-detection-adas-0001 from the intel model zoo and on few other device pedestrian-and-vehicle-detector-adas-0001.

- Robel

[JesusE_Intel]

Hi Robel,

Thank you for your patience, I have not been able to find a workaround for the first issue. I am currently trying to reproduce the XLINK errors, I will let you know what I find out.

Regards,

Jesus

[JesusE_Intel]

Hi Robel,

I have been running the object_detection_demo_ssd_async with the face-detection-adas-0001.xml for more than 5 hours and have not been able to see the XLINK errors. Could you please try to make the following changes and let me know if this changes what you are seeing?

• Connect the Intel Neural Compute Stick 2 to a powered USB Hub
• Use the OpenVINO toolkit for Raspbian OS pre-built package, follow the installation instructions here

Regards,

Jesus

[Sharma__Robel]

Hello Jesus,

Thanks for the help. I will try it out and let you know the status.

- Robel

[Sharma__Robel]

Hello Jesus,

Sorry for the delay as we were on CES 2020 as AnyConnect and with ASUS with their Tinker-Edge-T. We have bought one good tplink USB power hub which provides 1.5A on USB C.

[ 1037.477350] hub 2-1.1:1.0: USB hub found
[ 1037.478736] hub 2-1.1:1.0: 4 ports detected
[ 1037.663071] usb 1-1.1.1.3: new high-speed USB device number 10 using xhci_hcd
[ 1037.793844] usb 1-1.1.1.3: New USB device found, idVendor=03e7, idProduct=2485, bcdDevice= 0.01
[ 1037.793859] usb 1-1.1.1.3: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[ 1037.793872] usb 1-1.1.1.3: Product: Movidius MyriadX
[ 1037.793884] usb 1-1.1.1.3: Manufacturer: Movidius Ltd.
[ 1037.793895] usb 1-1.1.1.3: SerialNumber: 03e72485
[ 1042.429557] usb 1-1.1.1.3: USB disconnect, device number 10
[ 1043.613082] usb 1-1.1.4: new high-speed USB device number 11 using xhci_hcd
[ 1043.743806] usb 1-1.1.4: New USB device found, idVendor=03e7, idProduct=2485, bcdDevice= 0.01
[ 1043.743812] usb 1-1.1.4: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[ 1043.743818] usb 1-1.1.4: Product: Movidius MyriadX
[ 1043.743823] usb 1-1.1.4: Manufacturer: Movidius Ltd.
[ 1043.743827] usb 1-1.1.4: SerialNumber: 03e72485
[ 1101.123879] usb 1-1.1.4: USB disconnect, device number 11

But still seeing the issue,

Jan 14 04:09:59 raspberrypi streamlauncher[8275]: InferenceEngine:
Jan 14 04:09:59 raspberrypi streamlauncher[8275]:         API version ............ 2.1
Jan 14 04:09:59 raspberrypi streamlauncher[8275]:         Build .................. custom_2019_b0c5accaf8e0c0fac17aaef688478d7f391d05bb
Jan 14 04:09:59 raspberrypi streamlauncher[8275]:         Description ....... API
Jan 14 04:09:59 raspberrypi streamlauncher[8275]: [ INFO ] Loading plugin
Jan 14 04:09:59 raspberrypi streamlauncher[8275]: [ INFO ] Plugin load done
Jan 14 04:09:59 raspberrypi streamlauncher[8275]: [ INFO ] Loading network files: /root/StreamApp/models/pedestrian-and-vehicle-detector-adas-0001.xml
Jan 14 04:09:59 raspberrypi streamlauncher[8275]: [ INFO ] Batch size is forced to  1.
Jan 14 04:09:59 raspberrypi streamlauncher[8275]: [ INFO ] Checking that the inputs are as the demo expects
Jan 14 04:09:59 raspberrypi streamlauncher[8275]: [ INFO ] Checking that the outputs are as the demo expects
Jan 14 04:09:59 raspberrypi streamlauncher[8275]: Reading labels size: 3 <==> classes: 3
Jan 14 04:09:59 raspberrypi streamlauncher[8275]: [ INFO ] Loading model to the plugin
Jan 14 04:10:01 raspberrypi streamlauncher[8275]: E: [ncAPI] [      1720] [StreamApp] ncDeviceOpen:859        Device doesn't appear after boot

In some cases, it can load and still shows,

Jan 14 03:01:28 raspberrypi streamlauncher[343]: E: [xLink] [ 840017] [EventRead00Thr] eventReader:218 eventReader thread stopped (err -4)
Jan 14 03:01:28 raspberrypi streamlauncher[343]: E: [watchdog] [ 840018] [WatchdogThread] sendPingMessage:121 Failed send ping message: X_LINK_ERROR
Jan 14 03:01:28 raspberrypi streamlauncher[343]: E: [watchdog] [ 841017] [WatchdogThread] sendPingMessage:121 Failed send ping message: X_LINK_ERROR
Jan 14 03:01:28 raspberrypi streamlauncher[343]: E: [watchdog] [ 841017] [WatchdogThread] watchdog_routine:315 [0x16c31f0] device, not respond, removing from watchdog
Jan 14 03:01:28 raspberrypi streamlauncher[343]:

Now I will try the libs from your provided link.

FYI, I ran dmesg -w

And seen the dongle disconnect on  E: [ncAPI] [    790356] [StreamApp] ncDeviceOpen:859        Device doesn't appear after boot

[JesusE_Intel]

Hi Robel,

Please let me know the outcome of your tests with the pre-built package.

Regards,

Jesus

[Sharma__Robel]

Hi Jesus,

I tried the latest pre-built package. On RPi4 soft reboot, the device (Movidius) still fails to load model sometimes on second boot-up. But the occurrence is lower.

Thanks,

Robel

[JesusE_Intel]

Hi Robel,

We just released the latest OpenVINO toolkit 2020.1. Could you see if this release has a lower occurrence than your previous test?

https://download.01.org/opencv/2020/openvinotoolkit/

Regards,

Jesus

[Sharma__Robel]

Hi Jesus,

Thanks for the help.

I will deploy it and let you know. But the last fix is performing well (around 80% time).

Regards,

Robel

[Sharma__Robel]

Hi Jesus,

Just an update, looks like latest library requires libgraph. It is giving me linking error on my project. Anyway, I will resolve it and put the result soon.

- Robel