MultiStick + Multiprocessing + IECore

[Mas__Juan]

Hi,

I am currently working on a solution of face recognition which implements two different neural networks, one for identification and the other one for recognition. I want to use MultiStick with NCS2 using multiprocessing to implement the consumer/producer processes. Before 2019 R2, it all worked well, but when I started using IECore it stopped working. It always prints the same error:

Traceback (most recent call last):
  File "/usr/lib/python3.5/multiprocessing/process.py", line 249, in _bootstrap
    self.run()
  File "/usr/lib/python3.5/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/home/juan/PAI/tfm-openvino/video_face_detection.py", line 207, in launch_InferenceCase
    ncs_worker = InferenceCase(device, i)
  File "/home/juan/PAI/tfm-openvino/video_face_detection.py", line 38, in __init__
    self.face_recognition = models.FaceNet(device, self.recognizer_model_fn, RECOGNITION_THRESHOLD, DEFAULT_NUM_INFERS, ieCore)
  File "/home/juan/PAI/tfm-openvino/models.py", line 160, in __init__
    super().__init__(device, model_xml, detection_of, num_infers, ie)
  File "/home/juan/PAI/tfm-openvino/models.py", line 53, in __init__
    device, num_infers, ie)
  File "/home/juan/PAI/tfm-openvino/models.py", line 78, in _load_ir_to_plugin
    exec_net = ie.load_network(network=self.net, device_name=device, num_requests=num_infers)
  File "ie_api.pyx", line 85, in openvino.inference_engine.ie_api.IECore.load_network
  File "ie_api.pyx", line 92, in openvino.inference_engine.ie_api.IECore.load_network
RuntimeError: Can not init Myriad device: NC_ERROR

Testing by myself, I could see that the problem resides in the active hub I am using to connect the NCS2. When connected to different USB ports, it all works well, but when connected to the hub it doesn't work like before the update.

I am creating a single instance of IECore for all the processes but it fails when creating multiple executable networks. I tried printing the available devices to compare both cases. The results are the following:

Different USB ports:

Process 1 -> ['CPU', 'GNA', 'MYRIAD.3.10-ma2480', 'MYRIAD.3.3-ma2480']
Process 2 -> ['CPU', 'GNA', 'MYRIAD']

OK

Using HUB:

Process 1 -> ['CPU', 'GNA', 'MYRIAD.3.2.1-ma2480', 'MYRIAD.3.2.2-ma2480']

Process 2 -> ['CPU', 'GNA', 'MYRIAD']

FAILS

In each process I am creating two different executable networks.

Any help or advice would be extremely appreciated,

Best regards

[Shubha_R_Intel]

Dear Mas, Juan,

We have a similar thread on dldt issue 226 . Does this help you ?

Repeated here:

 could you check you usbfs setting - cat /sys/module/usbcore/parameters/usbfs_memory_mb, if it smaller than 15, even 3 sticks / 3 networks might fail. Typical value is 16 that enough for 3 devices. Also there might be other issues with usb data transfer - do you have any error log?

Thanks,

Shubha

[Mas__Juan]

Shubha R. (Intel) wrote:

Dear Mas, Juan,

We have a similar thread on dldt issue 226 . Does this help you ?

Repeated here:

 could you check you usbfs setting - cat /sys/module/usbcore/parameters/usbfs_memory_mb, if it smaller than 15, even 3 sticks / 3 networks might fail. Typical value is 16 that enough for 3 devices. Also there might be other issues with usb data transfer - do you have any error log?

Thanks,

Shubha

First, thanks for replying. Then, I checked usbfs settings and got 16 as memory size. Checking with dmesg, I could not see any error. ¿Could you please indicate where I can find more information about USB events? The behaviour with dmesg is similar using the hub and using different ports, but only with the hub the program fails.

I am using Ubuntu 16.04, sorry I didn't mention it before. Intel examples work fine with MYRIAD.

Best regards

[Mas__Juan]

Hi,

I have recently discovered this error in initializacion of MYRIAD resides on the allocation of the networks in the concrete devices. When using multiple networks on multiple devices, openvino allocates equally the networks as mentioned in this section of the documentation (Device allocation). When using multiprocessing, this become a little bit more complicated as sticks are not shared through the different processes. In my case the error came as explained in the example:

launch process 1:

    init network 1

    init network 2

launch process 2:

    . . .

Now, the openvino scheduler could allocate network 1 in device 1 and network 2 in device 2 for the first process, leaving the second process unable to use any of the sticks. The same case when it allocates network 1 from process 1 in device 1, and network 1 from process 2 in device 2. It depends on how you (and the CPU scheduler) execute the processes.

The solution seems now quite obvious, it is only needed to pass the full device name of device 1 to process 1 and the full name of device 2 in process 2. However, I found that, sometimes, when using MYRIAD with an active HUB the available devices list goes from this (['CPU', 'GNA', 'MYRIAD.3.2.3-ma2480', 'MYRIAD.3.2.4-ma2480']) to this (['CPU', 'GNA', 'MYRIAD']) even if there are the same number of devices; which is kind of strange and may cause the program to fail.

Anyone has an explanation to this last situation? Thanks in advance,

Juan Mas

[Shubha_R_Intel]

Dear Mas, Juan,

This seems like an OS/Driver related issue doesn't it ? Meaning

Active HUB the available devices list goes from this (['CPU', 'GNA', 'MYRIAD.3.2.3-ma2480', 'MYRIAD.3.2.4-ma2480']) to this (['CPU', 'GNA', 'MYRIAD'])

is not related to OpenVino.

But yes, it's a kernel/driver issue. Perhaps the below article will help you - it's about assigning persistent names to USB ports for a given device.

https://unix.stackexchange.com/questions/394411/persistent-names-for-physical-usb-ports

Hope it helps,

Shubha