benchmark demo segmentation fault

[ASant103]

Hello,

we have an AI core module  https://up-board.org/ai-edge/ai-core/ connected through PCIe to main Xeon Processor running Linux Centos.

In reality we can see Movidius MyriadX connected directly via USB 2.0 to root Hub onboard : 

/:  Bus 04.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/1p, 480M
    |__ Port 1: Dev 96, If 0, Class=Vendor Specific Class, Driver=, 480M   ( Movidius )

Bus 004 Device 096: ID 03e7:2485 Intel Movidius MyriadX
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               2.00
  bDeviceClass            0 (Defined at Interface level)
  bDeviceSubClass         0 
  bDeviceProtocol         0 
  bMaxPacketSize0        64
  idVendor           0x03e7 Intel
  idProduct          0x2485 Movidius MyriadX
  bcdDevice            0.01
  iManufacturer           1 Movidius Ltd.
  iProduct                2 Movidius MyriadX
  iSerial                 3 03e72485
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength           32
    bNumInterfaces          1
    bConfigurationValue     1
    iConfiguration          0 
    bmAttributes         0x80
 

We can usually run the benchmark demo OpenVINO application without problems but we sometimes see benchmark OpenVINO demo crashing with segmentation fault or other occasions where Movidius is not enumerated.

I can now trigger the segmentation fault with 100% success by continuously launching benchmark demo and stopping it with Ctrl-C. If I stop it near the very beginning and immediately restart it, after few attempts I get the Movidius and the benchmark demo completely stuck.

here is one of the log when we get into error :

[root@xxx]# /root/inference_engine_samples_build/intel64/Release/benchmark_app -i /root/inference_engine_samples_build/intel64/Release/car.png -m /root/squeezenet1.1_FP16/squeezenet1.1.xml -d MYRIAD
[ INFO ] InferenceEngine:
        API version ............ 1.6
        Build .................. custom_releases/2019/R1_c9b66a26e4d65bb986bb740e73f58c6e9e84c7c2

[Step 1/8] Parsing and validation of input args
[ INFO ] Parsing input parameters
[ INFO ] Files were added: 1
[ INFO ]     /root/inference_engine_samples_build/intel64/Release/car.png
Progress: [....................] 100.00% done

[Step 2/8] Loading plugin
[ INFO ]
        API version ............ 1.6
        Build .................. 22443
        Description ....... myriadPlugin
Progress: [....................] 100.00% done

[Step 3/8] Read IR network
[ INFO ] Loading network files
[ INFO ] Network batch size: 1, precision: FP32
Progress: [....................] 100.00% done

[Step 4/8] Configure input & output of the model
[ INFO ] Preparing output blobs
Progress: [....................] 100.00% done

[Step 5/8] Loading model to the plugin
I: [ncAPI] [    280563] ncDeviceOpen:737        ncDeviceOpen() XLinkBootRemote is running for 4.1-ma2480...

W: [ncAPI] [    285664] ncDeviceOpen:750        ncDeviceOpen() XLinkBootRemote returned error X_LINK_COMMUNICATION_FAIL for 4.1-ma2480
E: [ncAPI] [  253.882114] benchmark_app[2398]: segfault at 0 ip 00007f0a965fcb20 sp 00007ffe09c07968 error 4[    300664] ncD in libc-2.17.so[7f0a964bd000+1c3000]
eviceOpen:799   Failed connection to device (4.1-ma2480) with error 5
Segmentation fault
[root@xxxt]#

At this point we have seen at least in one occasion where ProductID and manufacturer appear with "Error" on output to lsusb command regarding Movidius. But other times Movidius still looks being there in the lsusb enumerated peripherals.

Once the system is in this state it is not possible to run again benchmark test correctly even after reboot command, that issues an entire reboot of the main processor operating system. 

After reboot command, Movidius is not enumerated as connected peripheral through USB and lsusb command does not list it anymore.

Of course at this point benchmark demo OpenVINO still crashes .

Only a power cycling the entire system allows to recover correct communication and allows to run again the benchmark OpenVINO demo successfully.

Do you have any suggestion or idea what we could try in order to make the system more robust and avoid this problem to occur ?

thank you

Kind Regards

[Luis_at_Intel]

Hi Santagiuliana, Antonio

Thanks for reaching out. This seems to be related to a known issue we are currently taking a look at, let me reach out to the development team to see if there is a workaround. 

Regards,

Luis