Intel® Distribution of OpenVINO™ Toolkit
Community support and discussions about the Intel® Distribution of OpenVINO™ toolkit, OpenCV, and all things computer vision-related on Intel® platforms.
5919 Discussions

How to use Multi Stick NCS2 (Parallelization of reasoning)

Hyodo__Katsuya
Innovator
2,136 Views

Hello. Expert engineers.

In the past I have created a multi-stick + multi-process high performance Python program using NCSDK2.
https://github.com/PINTO0309/MobileNet-SSD-RealSense.git

However, now I am working on migration from NCSDK2 to OpenVINO.
My environment is as follows.

- OpenVINO R5 2018.5.445
- RaspberryPi3 (Raspbian Stretch)
- OpenCV 4.0.1-openvino
- Python3.5
- Neural Compute Stick 2 (x 3 pieces)

According to the following article Intel engineers,
https://software.intel.com/en-us/forums/computer-vision/topic/798625

He said that we can use multiple sticks by writing below.

plugin = IEPlugin(device = "MYRIAD")
plugin.set_config({"VPU_FORCE_RESET": "NO"})

I tried it in my repository.
It operates normally.
https://github.com/PINTO0309/MobileNet-SSD-RealSense/blob/master/MultiSt.

However, connecting the three NCS 2 to the self-powered USB 3.0 hub did not improve performance at all. (9 FPS - 10 FPS)

1. What kind of implementation will I be able to use with multi stick?
2. "VPU_FORCE_RESET": "NO": Does it make sense to use?

RaspberryPi3 + NCS2 [9 FPS - 10 FPS] https://youtu.be/GJNkX-ZBuC8

Core i7 + NCS2 [21 FPS] https://youtu.be/1ogge90EuqI

0 Kudos
1 Solution
Yuanyuan_L_Intel
Employee
2,136 Views

@Hyodo, Katsuya

I wrote a very simple demo to show how to use multiple ncs2 devices for inference. You can refer to below link and there is a readme under the same folder.

https://github.com/yuanyuanli85/open_model_zoo/blob/ncs2/demos/python_demos/multiple_device_ncs2_asy...;

View solution in original post

12 Replies
Yuanyuan_L_Intel
Employee
2,136 Views

Hi, 

You can follow the "Multiple NCS Devices" section in https://software.intel.com/en-us/articles/transitioning-from-intel-movidius-neural-compute-sdk-to-op...; Basically,  multiple ExecutableNetwork instances need to be created.  Then, you can use those multiple executablenetwork instances (usually async api is used ) to accelerate your inference.

Hyodo__Katsuya
Innovator
2,136 Views
@Yuanyuan L. (Intel) Thanks for your reply. As you gave me advice, I referred to the following article. https://software.intel.com/en-us/articles/transitioning-from-intel-movidius-neural-compute-sdk-to-op... However, it contains only a very simple explanation and I do not know the implementation image. Please let me know if you have a sample program or URL that I will refer to.
Yuanyuan_L_Intel
Employee
2,137 Views

@Hyodo, Katsuya

I wrote a very simple demo to show how to use multiple ncs2 devices for inference. You can refer to below link and there is a readme under the same folder.

https://github.com/yuanyuanli85/open_model_zoo/blob/ncs2/demos/python_demos/multiple_device_ncs2_asy...;

Hyodo__Katsuya
Innovator
2,136 Views
@Yuanyuan L. (Intel) Thank you for having a clear answer. I finally understood how to implement it. I will immediately begin implementing "MultiProcess" instead of "MultiThread".
Truong__Dien_Hoa
New Contributor II
2,136 Views

Sorry for the very amateur question but what is the point for using MultiProcess instead of MultiThread in this case ? I have read some tutorials on pyimagesearch about MultiThreading on video processing but not MultiProcessing. Thank you in advance

 

Hyodo__Katsuya
Innovator
2,136 Views

@Truong, Dien Hoa

It is an easy implementation like a toy, but I hope you will be helpful.

Python + MultiProcess + MultiThread + MultiModel + NCS2

https://github.com/PINTO0309/OpenVINO-EmotionRecognition.git

Python + MultiProcess + Google Edge TPU Accelerator

https://github.com/PINTO0309/TPU-MobilenetSSD.git

Truong__Dien_Hoa
New Contributor II
2,136 Views

@Katsuya, Thank you. Actually, I ask the question because passing by your code on MobileNetSSD which I think quite same concept. I am curious on why you are using MultiProcess and not just MultiThread. You are a really good practionner !! Most of things I need I can find in your work :D

Hyodo__Katsuya
Innovator
2,136 Views

Thank you for giving me a warm word. I am very happy.

In "Python + MultiThread" there is a problem of "GIL".

Global Interpreter Lock

http://www.dabeaz.com/python/UnderstandingGIL.pdf

As long as you are using Python, you can not get the best performance with multiple parallel thread handling.

Your best choice,

1. Implement MultiProcess yourself

2. Introducing Python's optional module "Flask"

 

$ sudo apt install python3-pip -y
$ pip3 install flask

 

Truong__Dien_Hoa
New Contributor II
2,136 Views

Thank you for your response. I will read carefully your document.

Nguyen__Anh_Minh
Beginner
2,136 Views

Hyodo, Katsuya wrote:

@Yuanyuan L. (Intel)

Thank you for having a clear answer.
I finally understood how to implement it.
I will immediately begin implementing "MultiProcess" instead of "MultiThread".

Hello, I find your repo OpenVINO-YoloV3 very amazing and I am checking out the openvino_yolov3_MultiStick_test.py application on my 2 sticks.

I found that you still used the multithreading for the inferencer process. should it be affected by the GIL problem too? please help me broaden my horizon on this.

thanks

Hyodo__Katsuya
Innovator
2,136 Views

@Nguyen, Anh Minh

Use NCSDK for a complete MultiProcess + MultiStick implementation.

OpenVINO can not control each stick accurately.

Reply