NCS reading error after some time of proper working + unable to recover even after raspberry pi reboot

[UPetr]

I am using Movidius NCS (older version) with my Raspberry pi 3 B+ to improve inference performance.

Initially, I used the stick without any extension cable or self-powered USB hub. The whole system is powered using a car battery and some power supply, DC-DC converter 12V -> 5V/3A, in between - so it can be said it's a not so stable environment. After some time of proper functioning (it doesn't happen all the time btw) this read error occurs and is endlessly repeated until some timeout is reached (approx ~ 30 mins):

dispatcherEventReceive:308 dispatcherEventReceive() Read failed -4 | event 0x6aafde20 USB_READ_REL_REQ

eventReader:256 eventReader stopped

[watchdog] [  173586] sendPingMessage:164 Failed send ping message: X_LINK_ERROR

Initially I used OpenVino's async approach as suggested, but have switched to synchronous inference invocation (res = self.net.infer({'data' : image})) to be able to catch and handle this Exception.

Nevertheless, when it happens and the python program is restarted I always have an init error during network loading process:

plugin = IEPlugin(device=device, plugin_dirs=plugin_dirs)

graph = IENetwork(model=model_name_xml, weights=model_name_bin)

assertlen(graph.inputs.keys()) ==1, "Supports only single input topologie"

assertlen(graph.outputs) ==2, "Supports only double output topologie"

input_blob =next(iter(graph.inputs))

n, c, h, w =graph.inputs[input_blob].shape

net =plugin.load(network=graph, num_requests=1)

This time I also have a huge number of same printouts during ~ 30 min, and during that period I am not able to catch and handle this Exception, and my log is full with:

[watchdog] [ 187589] sendPingMessage:164 Failed send ping message: X_LINK_ERROR

NB - I also tried the same scenario with self-powered USB hub with the same result.

Can you help me solve this mystery?

[JesusE_Intel]

Hi UPetr,

The X_LINK_ERROR is usually seen when the Intel Neural Compute Stick is not receiving enough power. Do you see the same behavior when connecting the Raspberry Pi and powered USB Hub to a stable AC source?

Could you also share the model and inference code for me to test?

Regards,

Jesus

[UPetr]

Hi @Intel_Jesus​ , thanks for answering.

Yes, I could see the same behavior even when the NCS is connected through the same USB hub (Atolla self-powered 4-port USB 3.0 hub) but using stable AC source.

The biggest problem is that I cannot catch that exception and my system is frozen for round about 30-40 mins. During that time my log is full with:

[35mE: [watchdog] [ 703913] sendPingMessage:164 Failed send ping message: X_LINK_ERROR[0m

Even like this, it's not guaranteed to be catched and handled:

try: #self.net.start_async(request_id=0, inputs={self.input_blob: frame}) res = self.net.infer({'data' : frame}) except Exception: # HANDLE EXCEPTION

I noticed the rule of having 3 x 1.000.000 retries to read it correctly before the system is un-frozen, so the log is 3 x times like this:

[35mE: [watchdog] [ 810] sendPingMessage:164 Failed send ping message: X_LINK_ERROR[0m [35mE: [watchdog] [ 1810] sendPingMessage:164 Failed send ping message: X_LINK_ERROR[0m [35mE: [watchdog] [ 2810] sendPingMessage:164 Failed send ping message: X_LINK_ERROR[0m [35mE: [watchdog] [ 3810] sendPingMessage:164 Failed send ping message: X_LINK_ERROR[0m [35mE: [watchdog] [ 4810] sendPingMessage:164 Failed send ping message: X_LINK_ERROR[0m [35mE: [watchdog] [ 5811] sendPingMessage:164 Failed send ping message: X_LINK_ERROR[0m . . . [35mE: [watchdog] [ 996809] sendPingMessage:164 Failed send ping message: X_LINK_ERROR[0m [35mE: [watchdog] [ 997809] sendPingMessage:164 Failed send ping message: X_LINK_ERROR[0m [35mE: [watchdog] [ 998809] sendPingMessage:164 Failed send ping message: X_LINK_ERROR[0m [35mE: [watchdog] [ 999810] sendPingMessage:164 Failed send ping message: X_LINK_ERROR[0m

How can I catch it? Is there any default timeout I could change?

[JesusE_Intel]

Hi UPetr,

I would need to run some tests on my end, could you share your model and code to reproduce? Could you share some details of the model you are using?

• What framework and topology is it based on?
• What command did you use with the model optimizer to convert to IR?

Are you also seeing the same issue when running one of our sample demos or only with your code and model?

Regards,

Jesus

[UPetr]

Hi @Intel_Jesus​ ,

• It is based on OpenVino. I am using Intel's face-person-detection-retail-0002-fp16 pre-trained model.

Code:

def load_plugin(self, device, plugin_dirs=None): self.plugin = IEPlugin(device=device, plugin_dirs=plugin_dirs)   def load_net(self): self.graph = IENetwork(model=self.model_name_xml, weights=self.model_name_bin) assertlen(self.graph.inputs.keys()) ==1, "Supports only single input topologie" assertlen(self.graph.outputs) ==2, "Supports only double output topologie"   self.input_blob =next(iter(self.graph.inputs)) self.n, self.c, self.h, self.w =self.graph.inputs[self.input_blob].shape self.net =self.plugin.load(network=self.graph, num_requests=1)   def detect(self, frame, original_frame, w, h): faces_detected =None pedestrians_detected =Nonetry:   #self.net.start_async(request_id=0, inputs={self.input_blob: frame}) #self.net.requests[0].wait(-1) #faces_detected = self.net.requests[0].outputs['detection_out_face'] #pedestrians_detected = self.net.requests[0].outputs['detection_out_pedestrian']   # Use sync approach to be able to handle exception. In async mode error is just skipped. res =self.net.infer({'data' : frame}) faces_detected = res['detection_out_face'] pedestrians_detected = res ['detection_out_pedestrian']   except Exception: ...

So basically it's all your code + model.

[JesusE_Intel]

Hi UPetr,

Thanks for getting back to me, I did a quick test by modifying the object_detection_demo_ssd_async to inference with the face-person-detection-retail-0002-fp16 pre-trained model. After, 40-60 minutes of running, I did not see any X_LINK_ERROR messages.

Which version of the OpenVINO toolkit are you using? I tested with a Raspberry Pi 3 using the OpenVINO 2019 R2. If you have not updated already, please update to the latest (2019 R2) and let me know the outcome.

Also, please note that this pre-trained model has been deprecated in the latest release.

Regards,

Jesus