Intel® Distribution of OpenVINO™ Toolkit
Community assistance about the Intel® Distribution of OpenVINO™ toolkit, OpenCV, and all aspects of computer vision-related on Intel® platforms.

How to use Multi Stick NCS2 (Parallelization of reasoning)

Hyodo__Katsuya
Innovator
2,921 Views

Hello. Expert engineers.

In the past I have created a multi-stick + multi-process high performance Python program using NCSDK2.
https://github.com/PINTO0309/MobileNet-SSD-RealSense.git

However, now I am working on migration from NCSDK2 to OpenVINO.
My environment is as follows.

- OpenVINO R5 2018.5.445
- RaspberryPi3 (Raspbian Stretch)
- OpenCV 4.0.1-openvino
- Python3.5
- Neural Compute Stick 2 (x 3 pieces)

According to the following article Intel engineers,
https://software.intel.com/en-us/forums/computer-vision/topic/798625

He said that we can use multiple sticks by writing below.

plugin = IEPlugin(device = "MYRIAD")
plugin.set_config({"VPU_FORCE_RESET": "NO"})

I tried it in my repository.
It operates normally.
https://github.com/PINTO0309/MobileNet-SSD-RealSense/blob/master/MultiSt.

However, connecting the three NCS 2 to the self-powered USB 3.0 hub did not improve performance at all. (9 FPS - 10 FPS)

1. What kind of implementation will I be able to use with multi stick?
2. "VPU_FORCE_RESET": "NO": Does it make sense to use?

RaspberryPi3 + NCS2 [9 FPS - 10 FPS] https://youtu.be/GJNkX-ZBuC8

Core i7 + NCS2 [21 FPS] https://youtu.be/1ogge90EuqI

0 Kudos
1 Solution
Yuanyuan_L_Intel
Employee
2,921 Views

@Hyodo, Katsuya

I wrote a very simple demo to show how to use multiple ncs2 devices for inference. You can refer to below link and there is a readme under the same folder.

https://github.com/yuanyuanli85/open_model_zoo/blob/ncs2/demos/python_demos/multiple_device_ncs2_async.py ;

View solution in original post

0 Kudos
12 Replies
Yuanyuan_L_Intel
Employee
2,921 Views

Hi, 

You can follow the "Multiple NCS Devices" section in https://software.intel.com/en-us/articles/transitioning-from-intel-movidius-neural-compute-sdk-to-openvino-toolkit. ; Basically,  multiple ExecutableNetwork instances need to be created.  Then, you can use those multiple executablenetwork instances (usually async api is used ) to accelerate your inference.

0 Kudos
Hyodo__Katsuya
Innovator
2,921 Views
@Yuanyuan L. (Intel) Thanks for your reply. As you gave me advice, I referred to the following article. https://software.intel.com/en-us/articles/transitioning-from-intel-movidius-neural-compute-sdk-to-openvino-toolkit However, it contains only a very simple explanation and I do not know the implementation image. Please let me know if you have a sample program or URL that I will refer to.
0 Kudos
Yuanyuan_L_Intel
Employee
2,922 Views

@Hyodo, Katsuya

I wrote a very simple demo to show how to use multiple ncs2 devices for inference. You can refer to below link and there is a readme under the same folder.

https://github.com/yuanyuanli85/open_model_zoo/blob/ncs2/demos/python_demos/multiple_device_ncs2_async.py ;

0 Kudos
Hyodo__Katsuya
Innovator
2,921 Views
@Yuanyuan L. (Intel) Thank you for having a clear answer. I finally understood how to implement it. I will immediately begin implementing "MultiProcess" instead of "MultiThread".
0 Kudos
Truong__Dien_Hoa
New Contributor II
2,921 Views

Sorry for the very amateur question but what is the point for using MultiProcess instead of MultiThread in this case ? I have read some tutorials on pyimagesearch about MultiThreading on video processing but not MultiProcessing. Thank you in advance

 

0 Kudos
Hyodo__Katsuya
Innovator
2,921 Views

@Truong, Dien Hoa

It is an easy implementation like a toy, but I hope you will be helpful.

Python + MultiProcess + MultiThread + MultiModel + NCS2

https://github.com/PINTO0309/OpenVINO-EmotionRecognition.git

Python + MultiProcess + Google Edge TPU Accelerator

https://github.com/PINTO0309/TPU-MobilenetSSD.git

0 Kudos
Truong__Dien_Hoa
New Contributor II
2,921 Views

@Katsuya, Thank you. Actually, I ask the question because passing by your code on MobileNetSSD which I think quite same concept. I am curious on why you are using MultiProcess and not just MultiThread. You are a really good practionner !! Most of things I need I can find in your work :D

0 Kudos
Hyodo__Katsuya
Innovator
2,921 Views

Thank you for giving me a warm word. I am very happy.

In "Python + MultiThread" there is a problem of "GIL".

Global Interpreter Lock

http://www.dabeaz.com/python/UnderstandingGIL.pdf

As long as you are using Python, you can not get the best performance with multiple parallel thread handling.

Your best choice,

1. Implement MultiProcess yourself

2. Introducing Python's optional module "Flask"

 

$ sudo apt install python3-pip -y
$ pip3 install flask

 

0 Kudos
Truong__Dien_Hoa
New Contributor II
2,921 Views

Thank you for your response. I will read carefully your document.

0 Kudos
Nguyen__Anh_Minh
Beginner
2,921 Views

Hyodo, Katsuya wrote:

@Yuanyuan L. (Intel)

Thank you for having a clear answer.
I finally understood how to implement it.
I will immediately begin implementing "MultiProcess" instead of "MultiThread".

Hello, I find your repo OpenVINO-YoloV3 very amazing and I am checking out the openvino_yolov3_MultiStick_test.py application on my 2 sticks.

I found that you still used the multithreading for the inferencer process. should it be affected by the GIL problem too? please help me broaden my horizon on this.

thanks

0 Kudos
Hyodo__Katsuya
Innovator
2,921 Views

@Nguyen, Anh Minh

Use NCSDK for a complete MultiProcess + MultiStick implementation.

OpenVINO can not control each stick accurately.

0 Kudos
Reply