[NCS2, RPi] Running multiple models as a flask app

idata · ‎02-14-2019

I have a flask app running multiple models with one NCS2.

Implementation wise, individual models have their own module with initialization and inference functions that's imported to the main flask app.

There's a callback function that releases opencv resources, but it returns this error whenever I try to run a different model afterwards. (this one is called manually after running a face-detection model with flask)

E: [ncAPI] [    196800] resetAll:256     Failed to connect to stalled device, rc: X_LINK_ERROR
    Traceback (most recent call last):
      File "ssd.py", line 48, in <module>
        out = net.forward()
    cv2.error: OpenCV(4.0.1-openvino) /home/jenkins/workspace/OpenCV/OpenVINO/build/opencv/modules/dnn/src/op_inf_engine.cpp:553: error: (-215:Assertion failed) Failed to initialize Inference Engine backend: Can not init USB device: NC_DEVICE_NOT_FOUND in function 'initPlugin'

I tried resetting the USB manually to release the graph in the NCS2; but the moment I unplug the usb I get these messages. (also gonna try software reset with usbreset.c, but I expect something similar)

`

λ ~ python3 server/server

OpenCV(4.0.1-openvino) /home/jenkins/workspace/OpenCV/OpenVINO/build/opencv/modules/imgproc/src/resize.cpp:3784: error: (-215:Assertion failed) !ssize.empty() in function 'resize'

    //Unplugged and replugged NCS2 without calling the face detection function - A okay

    //Unplugged NCS2 after running it and stopping it
    E: [xLink] [    938622] dispatcherEventReceive:308      dispatcherEventReceive() Read failed -1 | event 0x67b01e00 USB_READ_REL_REQ

    E: [xLink] [    938622] eventReader:256 eventReader stopped
    E: [watchdog] [    938711] sendPingMessage:164  Failed send ping message: X_LINK_ERROR
    E: [watchdog] [    939711] sendPingMessage:164  Failed send ping message: X_LINK_ERROR
    E: [watchdog] [    940712] sendPingMessage:164  Failed send ping message: X_LINK_ERROR
    E: [watchdog] [    941712] sendPingMessage:164  Failed send ping message: X_LINK_ERROR
    E: [watchdog] [    942712] sendPingMessage:164  Failed send ping message: X_LINK_ERROR
    E: [watchdog] [    943712] sendPingMessage:164  Failed send ping message: X_LINK_ERROR
    E: [watchdog] [    944713] sendPingMessage:164  Failed send ping message: X_LINK_ERROR
    E: [watchdog] [    945713] sendPingMessage:164  Failed send ping message: X_LINK_ERROR
    E: [watchdog] [    946713] sendPingMessage:164  Failed send ping message: X_LINK_ERROR
    E: [watchdog] [    947713] sendPingMessage:164  Failed send ping message: X_LINK_ERROR
    E: [watchdog] [    948713] sendPingMessage:164  Failed send ping message: X_LINK_ERROR
    E: [watchdog] [    949714] sendPingMessage:164  Failed send ping message: X_LINK_ERROR
    ...
`

Module itself has this structure

import cv2
from importlib import reload

net = ""
cap = ""

def get_frame():
    ...
    return jpeg.tobytes()

def gen():
    ...
    yield (get_frame())

def cam_start():
    global cap, net

    #net = cv2.dnn.readNet('face-detection-adas-0001.xml', 'face-detection-adas-0001.bin')
    net = cv2.dnn.readNet('/home/pi/server/face_detection/face-detection-retail-0004.xml',
                          '/home/pi/server/face_detection/face-detection-retail-0004.bin')

    # Specify target device
    net.setPreferableTarget(cv2.dnn.DNN_TARGET_MYRIAD)
    cap = cv2.VideoCapture(0, cv2.CAP_V4L)
    cap.set(cv2.CAP_PROP_FPS, 15)
    cap.set(cv2.CAP_PROP_FRAME_WIDTH, 480)
    cap.set(cv2.CAP_PROP_FRAME_HEIGHT, 360)
    return "cam started"

def cam_stop():
    global cap, net
    del net
    net = ""
    cap.release()
    reload(cv2)
    #cv2.destroyAllWindows()
    return "cam stopped"

They both run fine individually.

I looked through opencv4.0.1 documentation to look for a way to release the graph or anything relevant in the dnn module section, but I've had no luck so far.

Anything I might've missed or hacky potential solution would be appreciated :)

Thanks,

idata · ‎02-19-2019

Hi @minswho ,

Thanks for contacting us. Let me know if I understand correctly, you have two models that run fine individually but when you attempt to use them together it fails, may I ask which release of OpenVINO Toolkit are you using? Also your host OS and any other details that could be helpful (i.e. powered USB hub 2.0/3.0, Ubuntu 16.04, etc.). If you could also share your code to help us reproduce the problem would be great.

Regards,

@Luis_at_Intel

idata · ‎02-28-2019

I believe I'm running R5. And no, models are run one at a time. However, when I try to load the NCS with another model after having already run one, it gives out the "failed to connect to stalled device" error, even with cam_stop() releasing opencv resources. I found a hacky workaround where I intentionally cause a module import error to kind of hard reset (idk, it's a throwaway demo under time crunch), but I'd still like to find out how I can "reset" the NCS2.

Finally, this is a bit unrelated, but can I ask if movidius plan on eventually supporting RNN with NCS2? If not, could you recommend any similar chipsets that support RNN who'll be in volume manufacturing by Q4 2019?

idata · ‎02-28-2019

@Luis_at_Intel

idata · ‎03-03-2019

@Luis_at_Intel I encountered exactly the same error message on my UP-BOARD computer (Win10 1709) with NCS2 (through additional power supply USB3.0 Y cable) and OpenVINO 2018 R5.0.1. I believe this issue can be easily reproduced with OpenVino TinyYOLO object detection Python example (async mode) after running the object detection 2~3 hours continuously. I have no idea why it's not stable under torture testing. Hopefully, NCS2 is designed for 24x7 workload, and it's not an overheating issue.

WinUsb_ReadPipe: System err 2

[35mE: [xLink] [ 0] handleIncomingEvent:240 handleIncomingEvent() Read failed -2

[0m

WinUsb_WritePipe: System err 22

[33mW: [xLink] [ 0] dispatcherEventReceive:324

WinUsb_WritePipe failed with error:=22

[35mE: [xLink] [ 0] dispatcherEventSend:889 Write failed header -2 | event USB_WRITE_REQ

[0m

Failed to handle incoming event[0m

WinUsb_SetPipePolicy: System err 22

[35mE: [xLink] [ 0] dispatcherEventReceive:308 dispatcherEventReceive() Read failed -2 | event 00000010C2B2FEA0 USB_WRITE_REQ

[0m

[35mE: [xLink] [ 0] eventReader:256 eventReader stopped[0m

[35mE: [ncAPI] [ 0] ncGraphQueueInference:3538 Can't send trigger request[0m

[35mE: [watchdog] [ 0] sendPingMessage:164 Failed send ping message: X_LINK_ERROR[0m

idata · ‎03-04-2019

@Luis_at_Intel BTW, the error I encountered seems to be related to https://ncsforum.movidius.com/discussion/1106/ncs-temperature-issue

I'm not able to reproduce the USB link disconnection issue when NCS2 running under sync mode (using infer() instead of start_async() ) with TinyYOLO Python example. Is it possible to return a meaningful error code rather than USB link disconnection while NCSv2 overheating?

idata · ‎03-19-2019

@Luis_at_Intel

Hi Luis, I just wanted to follow up.

Your input would be really appreciated.

idata · ‎03-19-2019

Hi @minswho ,

Apologies for the delay in my response. Unfortunately we cannot comment or share any roadmap information for the NCS/NCS2 devices, but I will provide this feedback to the engineering team. As far as any recommendations go for similar chipsets that support RNN I'm sorry but I don't have any insights for that.

I am trying to find out if there is a way in the OpenVINO toolkit to reset the device, I will get back to you as I get more information. Thank you for your patience.

Regards,

@Luis_at_Intel

idata · ‎03-22-2019

Hi @minswho ,

Thank you for your patience. I believe I have found information related on how to reset the NCS device within OpenVINO. There is a parameter KEY_VPU_FORCE_RESET that resets any stalled devices on plugin initialization, it must be used with SetConfig method. I am hoping to find the equivalent configuration parameters in python, but I think the information below should hopefully guide you in the right direction.

Myriad Plugin, Supported Configuration Parameters

https://docs.openvinotoolkit.org/2018_R5/_docs_IE_DG_supported_plugins_MYRIAD.html

VPU Plugins, VPU Common Configuration Parameters

https://docs.openvinotoolkit.org/2018_R5/_docs_IE_DG_supported_plugins_VPU.html

Regards,

@Luis_at_Intel