- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@Sahira_at_Intel
Could you do me a favor to find the root cause from the below error message? I got it from my UP-BOARD computer (Win10 1709) with NCS2 (through additional power supply USB3.0 Y cable) and OpenVINO 2018 R5.0.1.
My code is basically derived from the C:\Intel\computer_vision_sdk\inference_engine\samples\python_samples\object_detection_demo_yolov3.py sample code with a few modifications, and I saw the error with running it continuously 1~2 hours.
1) Receiving IPCAM streaming frames (30fps) as the input source rather than using Webcam in a dedicated Python worker thread (i.e. threading.Thread)
2) The IEPlugin and IENetwork related objects are all being instantiated/operated in that dedicated worker thread rather than the main thread.
3) Main thread is used for rendering the object detection result. It communicates with the worker thread through a synchronized queue.
I suspected it's related to NCS2 overheating issue but now I don't think so. Last night, I tried to isolate all IEPlugin and IENetwork related code into a Python worker process (i.e. multiprocessing.Process) and communicate with it through IPC (i.e. multiprocessing.Pipe). It's very weird that now I'm not able to reproduce the error message after running the job 8 hours. Haha!
As you've known there are some other people seeing the same error code in https://ncsforum.movidius.com/discussion/1106/ncs-temperature-issue
Could you still escalate the issue to RD core team? I'm wondering to know what the error message mean. Thank you.
WinUsb_ReadPipe: System err 2
[35mE: [xLink] [ 0] handleIncomingEvent:240 handleIncomingEvent() Read failed -2
[0m
WinUsb_WritePipe: System err 22
[33mW: [xLink] [ 0] dispatcherEventReceive:324
WinUsb_WritePipe failed with error:=22
[35mE: [xLink] [ 0] dispatcherEventSend:889 Write failed header -2 | event USB_WRITE_REQ
[0m
Failed to handle incoming event[0m
WinUsb_SetPipePolicy: System err 22
[35mE: [xLink] [ 0] dispatcherEventReceive:308 dispatcherEventReceive() Read failed -2 | event 00000010C2B2FEA0 USB_WRITE_REQ
[0m
[35mE: [xLink] [ 0] eventReader:256 eventReader stopped[0m
[35mE: [ncAPI] [ 0] ncGraphQueueInference:3538 Can't send trigger request[0m
[35mE: [watchdog] [ 0] sendPingMessage:164 Failed send ping message: X_LINK_ERROR[0m
[35mE: [watchdog] [ 0] sendPingMessage:164 Failed send ping message: X_LINK_ERROR[0m
[35mE: [watchdog] [ 0] sendPingMessage:164 Failed send ping message: X_LINK_ERROR[0m
[35mE: [watchdog] [ 0] sendPingMessage:164 Failed send ping message: X_LINK_ERROR[0m
[35mE: [watchdog] [ 0] sendPingMessage:164 Failed send ping message: X_LINK_ERROR[0m
[35mE: [watchdog] [ 0] sendPingMessage:164 Failed send ping message: X_LINK_ERROR[0m
[35mE: [watchdog] [ 0] sendPingMessage:164 Failed send ping message: X_LINK_ERROR[0m
- Tags:
- Assert
Link Copied
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Unfortunately, the error still can be reproduced after running 12 hours even isolating all IEPlugin and IENetwork related code into a Python worker process (i.e. multiprocessing.Process) and communicate with it through IPC (i.e. multiprocessing.Pipe).
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi @lilohuang
I was able to run a test on this issue. My setup: used an Intel NUC (Hades Canyon NUC8I7HVK, w/ 32 GB memory). running windows 10, and openvino R5.0.1. I ran the object detection for 4 - 5 hours and did not run into that error. The internal temperature of the NCS2 was ~50 C (maximum temperature is ~70 C for reference). So, I don't think this error is due to overheating. You said you were getting additional power through a USB y cable, so perhaps the cable is faulty? (That's what the Failed send ping message: X_LINK_ERROR[0m is telling me). Are you able to try on a different system and plug straight into the USB port if the UP board doesn't supply enough power by itself?
Best Regards,
Sahira
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@Sahira_at_Intel
Hi, I got 2 similar problems.
- While I'm using NCS2 to do face recognition, I touched the NCS2 to feel the temperature and got an electric shock, and similar warning message showed as @lilohuang got. Is there any problem with ESD…?
- Running the face recognition in async mode continuously 2 hours and it returns nothing. https://software.intel.com/en-us/forums/computer-vision/topic/805853
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@Sahira_at_Intel In fact, I've bought a new USB HUB with additional AC power instead of Y-cable after March, 9. Unfortunately, the error is still there with long running (>12 hours, 30 fps, async mode). I suspect it's a software bug (MYRIAD driver bug?) rather than hardware overheating issue. May I know how you get the internal temperature of the NCS2? through an internal debugging flag?
Sorry, I'm a programmer mainly on Mac OS X, I only have one UP-BOARD system to run the stress test. Of course, I will consider to spend more money to buy a INTEL NUC same as your spec if it indeed can fix the issue. If the power supply of NCS2 is critical, and then it conflicts with my understanding of Intel promotion video which claims NCS2 only requiring low power consumption.
Could you try to run TinyYOLOv3 (30 fps, async mode) over than 24 hours? Reading another post to see some update from other people. https://software.intel.com/en-us/forums/computer-vision/topic/805853 I believe I'm not the only one person suffered this problem. Thanks for your strong support!
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi @lilohuang @FREEMAN123
I have been running the object_detection_demo_yolov3_async on Ubuntu 16.04 with OpenVINO R5.0.1 for 24 hours now. When I started the demo, I hit the tab key to start "True Async", are you doing the same? My Neural Compute Stick 2 is connected to a powered USB Hub staying around 48 C (Temperature taken with a IR Thermometer).
Could you share a link to which yolo v3 model you are using? Are you using the tiny or full yolo model?
Regards,
Jesus
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@Jesus_at_Intel Thank you for your response. As I mentioned earlier, my code is basically derived from C:\Intel\computer_vision_sdk\inference_engine\samples\python_samples\object_detection_demo_yolov3.py from OpenVINO R5.0.1 rather than Intel OpenVino YOLOv3 C++ sample code. A few code modification on object_detection_demo_yolov3.py is required for parsing TinyYOLOv3 result, the model file is simply converted from a regular TinyYOLOv3 weight file for MYRIAD (FP16).
I can also reproduce the error through PINTO's repo. https://github.com/PINTO0309/OpenVINO-YoloV3 , using the converted model by PINTO under lrmodels/tiny-YoloV3/FP16/, and running openvino_tiny-yolov3_test.py even with synchronous mode (i.e. calling exec_net.infer()) on Windows 10 (1709) after running 12~16 hours. Of course, using Asynchronous mode (i.e. calling exec_net.start_async()) like Intel's object_detection_demo_yolov3.py did can increase the error reproducibility.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Today, I got the same error by using C:\Intel\computer_vision_sdk_2018.5.456\deployment_tools\inference_engine\samples\python_samples\object_detection_demo_ssd_async.py with pretrained model C:\Intel\computer_vision_sdk_2018.5.456\deployment_tools\intel_models\person-vehicle-bike-detection-crossroad-0078\FP16\person-vehicle-bike-detection-crossroad-0078.xml on my UP-BOARD with NCS2 after few hours. I believe INTEL NCS2 is NOT ONLY compatible with INTEL NUC. Could your team find some OTHER computers to test NCS2 device rather than INTEL NUC? Thanks.
[31mF: [xLink] [ 0] dispatcherEventReceive:318 WDuplicate id detected.
in[0m
Usb_WritePipe: System err 995
W
WinUsb_WritePipe failed with error:=995
i[35mE: [xLink] [ 0] dispatcherEventSend:908 nUsb_ReadPipeWrite failed event -2
:[0m
[31mF: [xLink] [ 0] dispatcherResponseServe:346 Sno request for this response: USB_READ_REL_RESP 1 9121
[0m
y#### (i == MAX_EVENTS) USB_READ_REL_RESP 1 9121
s0) header->id 9098, header->type USB_WRITE_REQ(0), curr->lQueue.q[i].isServed 4, EVENT_PENDING 1
tem1) header->id 9099, header->type USB_WRITE_REQ(0), curr->lQueue.q[i].isServed 4, EVENT_PENDING 1
2) header->id 9100, header->type USB_READ_REQ(1), curr->lQueue.q[i].isServed 4, EVENT_PENDING 1
3) header->id 9101, header->type USB_READ_REL_REQ(2), curr->lQueue.q[i].isServed 4, EVENT_PENDING 1
4) header->id 9102, header->type USB_READ_REL_REQ(2), curr->lQueue.q[i].isServed 4, EVENT_PENDING 1
5) header->id 9103, header->type USB_READ_REQ(1), curr->lQueue.q[i].isServed 4, EVENT_PENDING 1
6) header->id 9104, header->type USB_WRITE_REQ(0), curr->lQueue.q[i].isServed 4, EVENT_PENDING 1
7) header->id 9105, header->type USB_WRITE_REQ(0), curr->lQueue.q[i].isServed 4, EVENT_PENDING 1
8) header->id 9106, header->type USB_WRITE_REQ(0), curr->lQueue.q[i].isServed 4, EVENT_PENDING 1
9) header->id 9107, header->type USB_READ_REL_REQ(2), curr->lQueue.q[i].isServed 4, EVENT_PENDING 1
10) header->id 9108, header->type USB_READ_REQ(1), curr->lQueue.q[i].isServed 4, EVENT_PENDING 1
11) header->id 9109, header->type USB_READ_REL_REQ(2), curr->lQueue.q[i].isServed 4, EVENT_PENDING 1
12) header->id 9110, header->type USB_READ_REQ(1), curr->lQueue.q[i].isServed 4, EVENT_PENDING 1
e13) header->id 9111, header->type USB_WRITE_REQ(0), curr->lQueue.q[i].isServed 4, EVENT_PENDING 1
r14) header->id 9112, header->type USB_WRITE_REQ(0), curr->lQueue.q[i].isServed 4, EVENT_PENDING 1
r15) header->id 9113, header->type USB_READ_REQ(1), curr->lQueue.q[i].isServed 4, EVENT_PENDING 1
16) header->id 9114, header->type USB_READ_REL_REQ(2), curr->lQueue.q[i].isServed 4, EVENT_PENDING 1
2217) header->id 9115, header->type USB_READ_REL_REQ(2), curr->lQueue.q[i].isServed 4, EVENT_PENDING 1
18) header->id 9116, header->type USB_READ_REQ(1), curr->lQueue.q[i].isServed 4, EVENT_PENDING 1
19) header->id 9117, header->type USB_WRITE_REQ(0), curr->lQueue.q[i].isServed 4, EVENT_PENDING 1
20) header->id 9118, header->type USB_WRITE_REQ(0), curr->lQueue.q[i].isServed 4, EVENT_PENDING 1
21) header->id 9119, header->type USB_READ_REL_REQ(2), curr->lQueue.q[i].isServed 4, EVENT_PENDING 1
22) header->id 9120, header->type USB_READ_REQ(1), curr->lQueue.q[i].isServed 4, EVENT_PENDING 1
23) header->id 9121, header->type USB_READ_REL_REQ(2), curr->lQueue.q[i].isServed 4, EVENT_PENDING 1
24) header->id 9122, header->type USB_READ_REQ(1), curr->lQueue.q[i].isServed 2, EVENT_PENDING 1
25) header->id 9123, header->type USB_WRITE_REQ(0), curr->lQueue.q[i].isServed 1, EVENT_PENDING 1
26) header->id 9060, header->type USB_READ_REQ(1), curr->lQueue.q[i].isServed 4, EVENT_PENDING 1
27) header->id 9061, header->type USB_WRITE_REQ(0), curr->lQueue.q[i].isServed 4, EVENT_PENDING 1
28) header->id 9062, header->type USB_WRITE_REQ(0), curr->lQueue.q[i].isServed 4, EVENT_PENDING 1
29) header->id 9063, header->type USB_READ_REQ(1), curr->lQueue.q[i].isServed 4, EVENT_PENDING 1
30) header->id 9064, header->type USB_READ_REL_REQ(2), curr->lQueue.q[i].isServed 4, EVENT_PENDING 1
31) header->id 9065, header->type USB_READ_REL_REQ(2), curr->lQueue.q[i].isServed 4, EVENT_PENDING 1
32) header->id 9066, header->type USB_READ_REQ(1), curr->lQueue.q[i].isServed 4, EVENT_PENDING 1
33) header->id 9067, header->type USB_WRITE_REQ(0), curr->lQueue.q[i].isServed 4, EVENT_PENDING 1
34) header->id 9068, header->type USB_WRITE_REQ(0), curr->lQueue.q[i].isServed 4, EVENT_PENDING 1
[35mE: [xLink] [ 0] dispatcherEventReceive:308 dispatcherEventReceive() Read failed -2 | event 0000005077F4F730 USB_READ_REL_RESP
[0m
35) header->id 9069, header->type USB_READ_REQ(1), curr->lQueue.q[i].isServed 4, EVENT_PENDING 1
36) header->id 9070, header->type USB_READ_REL_REQ(2), curr->lQueue.q[i].isServed 4, EVENT_PENDING 1
37) header->id 9071, header->type USB_READ_REL_REQ(2), curr->lQueue.q[i].isServed 4, EVENT_PENDING 1
38) header->id 9072, header->type USB_READ_REQ(1), curr->lQueue.q[i].isServed 4, EVENT_PENDING 1
39) header->id 9073, header->type USB_WRITE_REQ(0), curr->lQueue.q[i].isServed 4, EVENT_PENDING 1
40) header->id 9074, header->type USB_WRITE_REQ(0), curr->lQueue.q[i].isServed 4, EVENT_PENDING 1
41) header->id 9075, header->type USB_READ_REQ(1), curr->lQueue.q[i].isServed 4, EVENT_PENDING 1
42) header->id 9076, header->type USB_READ_REL_REQ(2), curr->lQueue.q[i].isServed 4, EVENT_PENDING 1
43) header->id 9077, header->type USB_WRITE_REQ(0), curr->lQueue.q[i].isServed 4, EVENT_PENDING 1
44) header->id 9078, header->type USB_READ_REL_REQ(2), curr->lQueue.q[i].isServed 4, EVENT_PENDING 1
45) header->id 9079, header->type USB_READ_REQ(1), curr->lQueue.q[i].isServed 4, EVENT_PENDING 1
46) header->id 9080, header->type USB_WRITE_REQ(0), curr->lQueue.q[i].isServed 4, EVENT_PENDING 1
47) header->id 9081, header->type USB_WRITE_REQ(0), curr->lQueue.q[i].isServed 4, EVENT_PENDING 1
48) header->id 9082, header->type USB_READ_REQ(1), curr->lQueue.q[i].isServed 4, EVENT_PENDING 1
49) header->id 9083, header->type USB_READ_REL_REQ(2), curr->lQueue.q[i].isServed 4, EVENT_PENDING 1
50) header->id 9084, header->type USB_READ_REL_REQ(2), curr->lQueue.q[i].isServed 4, EVENT_PENDING 1
51) header->id 9085, header->type USB_READ_REQ(1), curr->lQueue.q[i].isServed 4, EVENT_PENDING 1
52) header->id 9086, header->type USB_WRITE_REQ(0), curr->lQueue.q[i].isServed 4, EVENT_PENDING 1
53) header->id 9087, header->type USB_WRITE_REQ(0), curr->lQueue.q[i].isServed 4, EVENT_PENDING 1
54) header->id 9088, header->type USB_READ_REQ(1), curr->lQueue.q[i].isServed 4, EVENT_PENDING 1
55) header->id 9089, header->type USB_READ_REL_REQ(2), curr->lQueue.q[i].isServed 4, EVENT_PENDING 1
56) header->id 9090, header->type USB_READ_REL_REQ(2), curr->lQueue.q[i].isServed 4, EVENT_PENDING 1
57) header->id 9091, header->type USB_READ_REQ(1), curr->lQueue.q[i].isServed 4, EVENT_PENDING 1
58) header->id 9092, header->type USB_WRITE_REQ(0), curr->lQueue.q[i].isServed 4, EVENT_PENDING 1
59) header->id 9093, header->type USB_WRITE_REQ(0), curr->lQueue.q[i].isServed 4, EVENT_PENDING 1
60) header->id 9094, header->type USB_READ_REL_REQ(2), curr->lQueue.q[i].isServed 4, EVENT_PENDING 1
61) header->id 9095, header->type USB_READ_REQ(1), curr->lQueue.q[i].isServed 4, EVENT_PENDING 1
62) header->id 9096, header->type USB_READ_REL_REQ(2), curr->lQueue.q[i].isServed 4, EVENT_PENDING 1
63) header->id 9097, header->type USB_READ_REQ(1), curr->lQueue.q[i].isServed 4, EVENT_PENDING 1
[35mE: [xLink] [ 0] eventReader:256 eventReader stopped[0m
[35mE: [ncAPI] [ 0] ncGraphQueueInference:3538 Can't send trigger request[0m
[35mE: [watchdog] [ 0] sendPingMessage:164 Failed send ping message: X_LINK_ERROR[0m
[35mE: [watchdog] [ 0] sendPingMessage:164 Failed send ping message: X_LINK_ERROR[0m
[35mE: [watchdog] [ 0] sendPingMessage:164 Failed send ping message: X_LINK_ERROR[0m
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi @lilohuang
I have been running the object_detection_demo_ssd_async.py
with the person-vehicle-bike-detection-crossroad-0078.xml
model on the Up Board. So far the sample has been running for 24 hours without the error message. Could you try to use a different powered USB Hub?
Regards,
Jesus
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@Jesus_at_Intel
Yes, the issue seems to be resolved after I bought a brand new 7 ports AC powered USB hub ($40 USD) instead of using my previous 4 ports AC powered USB hub ($30 USD). The brand new 7 ports AC powered USB hub has a LARGER AC adapter comparing to the 4 ports AC powered USB hub.
I’m not 100% sure, but the X_LINK_ERROR seems to be related to the USB controller chip compatibility or USB power system stability. By using the brand new 7 ports AC powered USB hub, it can pass 4 hours torture test (pre-trained TinyYOLOv3 model, 30fps camera input, async mode, num_requests = 2). I will keep running the torture testing in the following days, I expect NCS2 can work on 24x7 workload if the ambient temperature is controlled under 40˚C.
Perhaps, the NCS2 peak power consumption is larger than my understanding. Well.
Thank you so much!
Lilo
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Just an update, the X_LINK_ERROR doesn't occur any more after running 48 hours torture test. I think using a powerful AC powered USB hub is the right solution. Thanks.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi @lilohuang
Great! Thank you for confirming.
Regards,
Jesus
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Thanks mdata for your response, can you please confirm that you faced this issue again throughout this whole time?
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page