How to use Multi Stick NCS2 (Parallelization of reasoning)

[Hyodo__Katsuya]

Hello. Expert engineers.

In the past I have created a multi-stick + multi-process high performance Python program using NCSDK2.
https://github.com/PINTO0309/MobileNet-SSD-RealSense.git

However, now I am working on migration from NCSDK2 to OpenVINO.
My environment is as follows.

- OpenVINO R5 2018.5.445
- RaspberryPi3 (Raspbian Stretch)
- OpenCV 4.0.1-openvino
- Python3.5
- Neural Compute Stick 2 (x 3 pieces)

According to the following article Intel engineers,
https://software.intel.com/en-us/forums/computer-vision/topic/798625

He said that we can use multiple sticks by writing below.

plugin = IEPlugin(device = "MYRIAD")
plugin.set_config({"VPU_FORCE_RESET": "NO"})

I tried it in my repository.
It operates normally.
https://github.com/PINTO0309/MobileNet-SSD-RealSense/blob/master/MultiSt.

However, connecting the three NCS 2 to the self-powered USB 3.0 hub did not improve performance at all.　(9 FPS - 10 FPS)

1. What kind of implementation will I be able to use with multi stick?
2. "VPU_FORCE_RESET": "NO": Does it make sense to use?

RaspberryPi3 + NCS2 [9 FPS - 10 FPS] https://youtu.be/GJNkX-ZBuC8

Core i7 + NCS2 [21 FPS] https://youtu.be/1ogge90EuqI

[Yuanyuan_L_Intel]

Hi, 

You can follow the "Multiple NCS Devices" section in https://software.intel.com/en-us/articles/transitioning-from-intel-movidius-neural-compute-sdk-to-openvino-toolkit. ; Basically,  multiple ExecutableNetwork instances need to be created.  Then, you can use those multiple executablenetwork instances (usually async api is used ) to accelerate your inference.

[Hyodo__Katsuya]

@Yuanyuan L. (Intel) Thanks for your reply. As you gave me advice, I referred to the following article. https://software.intel.com/en-us/articles/transitioning-from-intel-movidius-neural-compute-sdk-to-openvino-toolkit However, it contains only a very simple explanation and I do not know the implementation image. Please let me know if you have a sample program or URL that I will refer to.

[Yuanyuan_L_Intel · Accepted solution]

@Hyodo, Katsuya

I wrote a very simple demo to show how to use multiple ncs2 devices for inference. You can refer to below link and there is a readme under the same folder.

https://github.com/yuanyuanli85/open_model_zoo/blob/ncs2/demos/python_demos/multiple_device_ncs2_async.py ;

[votecoffee]

The link wasn't valid for some reason. Here is the fixed link: https://github.com/yuanyuanli85/open_model_zoo/blob/ncs2/demos/python_demos/multiple_device_ncs2_async.py

[Hyodo__Katsuya]

@Yuanyuan L. (Intel) Thank you for having a clear answer. I finally understood how to implement it. I will immediately begin implementing "MultiProcess" instead of "MultiThread".

[Truong__Dien_Hoa]

Sorry for the very amateur question but what is the point for using MultiProcess instead of MultiThread in this case ? I have read some tutorials on pyimagesearch about MultiThreading on video processing but not MultiProcessing. Thank you in advance

[Hyodo__Katsuya]

@Truong, Dien Hoa

It is an easy implementation like a toy, but I hope you will be helpful.

Python + MultiProcess + MultiThread + MultiModel + NCS2

https://github.com/PINTO0309/OpenVINO-EmotionRecognition.git

Python + MultiProcess + Google Edge TPU Accelerator

https://github.com/PINTO0309/TPU-MobilenetSSD.git

[Truong__Dien_Hoa]

@Katsuya, Thank you. Actually, I ask the question because passing by your code on MobileNetSSD which I think quite same concept. I am curious on why you are using MultiProcess and not just MultiThread. You are a really good practionner !! Most of things I need I can find in your work :D

[Hyodo__Katsuya]

Thank you for giving me a warm word. I am very happy.

In "Python + MultiThread" there is a problem of "GIL".

Global Interpreter Lock

http://www.dabeaz.com/python/UnderstandingGIL.pdf

As long as you are using Python, you can not get the best performance with multiple parallel thread handling.

Your best choice,

1. Implement MultiProcess yourself

2. Introducing Python's optional module "Flask"

$ sudo apt install python3-pip -y
$ pip3 install flask

[Truong__Dien_Hoa]

Thank you for your response. I will read carefully your document.

[Nguyen__Anh_Minh]

Hyodo, Katsuya wrote:

@Yuanyuan L. (Intel)

Thank you for having a clear answer.
I finally understood how to implement it.
I will immediately begin implementing "MultiProcess" instead of "MultiThread".

Hello, I find your repo OpenVINO-YoloV3 very amazing and I am checking out the openvino_yolov3_MultiStick_test.py application on my 2 sticks.

I found that you still used the multithreading for the inferencer process. should it be affected by the GIL problem too? please help me broaden my horizon on this.

thanks

[Hyodo__Katsuya]

@Nguyen, Anh Minh

Use NCSDK for a complete MultiProcess + MultiStick implementation.

OpenVINO can not control each stick accurately.