UP AI Core X: Failed send ping message: X_LINK_ERROR

[Carlyon__Shane]

I am using UP AI Core X on Ubuntu 16.04 - https://up-shop.org/featured/261-up-ai-core-x.html

When I inference SSD Inception V2 model from the webcam input, I get these logs after a while:(10 min or so)

E: [xLink] [    450217] dispatcherEventReceive:368      dispatcherEventReceive() Read failed -4 | event 0x7f0e9dd00ee0 XLINK_READ_REL_REQ

E: [xLink] [    450217] eventReader:230 eventReader stopped
E: [xLink] [    450218] XLinkReadDataWithTimeOut:1377   Event data is invalid
E: [ncAPI] [    450218] ncFifoReadElem:3313     Packet reading is failed.
E: [watchdog] [    450386] sendPingMessage:132  Failed send ping message: X_LINK_ERROR
E: [watchdog] [    451385] sendPingMessage:132  Failed send ping message: X_LINK_ERROR
E: [watchdog] [    452387] sendPingMessage:132  Failed send ping message: X_LINK_ERROR
E: [watchdog] [    453387] sendPingMessage:132  Failed send ping message: X_LINK_ERROR
E: [watchdog] [    454389] sendPingMessage:132  Failed send ping message: X_LINK_ERROR
E: [watchdog] [    455388] sendPingMessage:132  Failed send ping message: X_LINK_ERROR
E: [watchdog] [    456387] sendPingMessage:132  Failed send ping message: X_LINK_ERROR
E: [watchdog] [    457386] sendPingMessage:132  Failed send ping message: X_LINK_ERROR
E: [watchdog] [    458386] sendPingMessage:132  Failed send ping message: X_LINK_ERROR
E: [watchdog] [    459385] sendPingMessage:132  Failed send ping message: X_LINK_ERROR
E: [watchdog] [    460385] sendPingMessage:132  Failed send ping message: X_LINK_ERROR
E: [watchdog] [    461384] sendPingMessage:132  Failed send ping message: X_LINK_ERROR
E: [watchdog] [    462384] sendPingMessage:132  Failed send ping message: X_LINK_ERROR
E: [watchdog] [    462385] watchdog_routine:327 [0x7f0e98086e00] device, not respond, removing from watchdog

For NCS, this seems to be the power issue. But I am not sure how I can resolve the power issue as the board is plugged in the mini-PCI-e slot of my PC.

[Sahira_Intel]

Hi Shane,

I apologize for the delay in my response. I was able to run SSD Inception V2 and have not gotten a X_LINK error so far. I'm going to continue to run this for the next few hours to verify it'll run for an extended period of time. 

I ran this with OpenVINO's object_detection_demo_ssd_async because I wasn't sure what you ran it with. If you can provide me with your model, I can try it on my end and see if I'm getting the same error. 

Best Regards,

Sahira 

[Carlyon__Shane]

Hi, Sahira.

Thanks for your reply!

Yeah, I have been using SSD_Inception_V2 model using object_detection_demo_ssd_async.py.

Cheers.

[Carlyon__Shane]

@Sahira.

Now, I am getting this error:

E: [xLink] [    446330] dispatcherWaitEventComplete:709     waiting is timeout, sending reset remote event
E: [xLink] [    451328] dispatcherEventSend:945     Write failed header -7 | event XLINK_PING_REQ

E: [xLink] [    451329] eventSchedulerRun:585       Event sending failed
E: [xLink] [    451330] dispatcherEventReceive:368  dispatcherEventReceive() Read failed -4 | event 0x7f7ce4844ee0 XLINK_WRITE_REQ

E: [xLink] [    451330] eventReader:230     eventReader stopped
E: [ncAPI] [    451330] resetAll:348         Failed to connect to stalled device, rc: X_LINK_TIMEOUT
E: [xLink] [    451330] dispatcherEventSend:945     Write failed header -4 | event XLINK_RESET_REQ

E: [xLink] [    451330] eventSchedulerRun:585       Event sending failed

The UP AI Core X is plugged in the mini-PCI-e.

Not sure why it failed to detect?

[Sahira_Intel]

Hi Shane,

What version of OpenVINO are you using? Below is how I converted the model and ran inference. Can you verify that your model was converted the same way. 

I converted the model using the Model Optimizer with this command: 

python3 mo_tf.py --input_model <path_to_model> --tensorflow_use_custom_operations_config <path_to_ssd_v2_support.json> --tensorflow_object_detection_api_pipeline_config <path_to_config_file> --data_type FP16

To run inference on the NCS2, i used this command:

./object_detecton_demo_ssd_async -i cam -m <path_to_frozen_xml> -d MYRIAD

Best Regards,

Sahira

[Carlyon__Shane]

@Sahira

- OpenVINO version: 2019.1.133

- Yeah, I use those python scripts to convert original inception v2 model.

This model has been working well for 6 months on NCS.

We switched to UP AI Core X a month ago and found this issue after running the engine for over an hour or so.

Is this due to the high temperature? I think it's pretty hot in Switzerland nowadays?

Cheers.

[Dobai__Lorand]

Hello,

@Shane, @Sahira Any updates on this?

I am encountering the exact same issue (X_LINK_ERROR) trying to run an SSD model on the Myriad X connected through mPCIe. The model runs fine when using CPU or NCS2 (USB interface). 

Thanks!

[Sahira_Intel]

Hi Shane,

I apologize for my late response. 

I don't think that environment temperature is the issue here. The maximum internal temperature for the NCS is about 70°C, so as long as the NCS is not too hot, it should not be causing an disconnect error. Are you able to run inference on a different system to see if this is specific to the board you're using?

Best Regards,

Sahira 

[ZAhme2]

Hi Sahira

Any fix for this yet I am facing the exact same issue?

E: [xLink] [    638396] [EventRead00Thr] handleIncomingEvent:267        handleIncomingEvent() Read failed (err -7)

E: [xLink] [    638624] [Scheduler00Thr] dispatcherEventSend:923        Write failed (event) (err -7)

E: [xLink] [    639664] [EventRead00Thr] dispatcherEventReceive:336     dispatcherEventReceive() Read failed (err -4) | event 0x7f37c37fde40 XLINK_WRITE_REQ

E: [xLink] [    639664] [EventRead00Thr] eventReader:223        eventReader thread stopped (err -4)
E: [xLink] [    639665] [python] XLinkReadDataWithTimeOut:1323  E: [watchdog] [    639665] [WatchdogThread] sendPingMessage:120 Failed send ping message: X_LINK_ERROREvent data is invalid
E: [ncAPI] [    639667] [python] checkGraphMonitorResponse:1762 XLink error, rc: X_LINK_ERROR
E: [ncAPI] [    639667] [python] ncGraphQueueInference:3914     Can't get trigger response

terminate called after throwing an instance of 'InferenceEngine::details::InferenceEngineException'
  what():  Failed to queue inference: NC_ERROR
Aborted (core dumped)