Intel® Distribution of OpenVINO™ Toolkit
Community assistance about the Intel® Distribution of OpenVINO™ toolkit, OpenCV, and all aspects of computer vision-related on Intel® platforms.
6392 Discussions

RealSense D435 + Neural Compute Stick + MobileNet-SSD + RaspberryPi3

idata
Employee
2,590 Views

It may already be released by someone, but I made a cooperative sample of RealSense D435 + Neural Compute Stick + MobileNet-SSD + RaspberryPi3.

 

Distance can also be measured while detecting objects.

 

I'm glad if you can use it as a reference.

 

This time, it is single stick + single thread.

 

https://github.com/PINTO0309/MobileNet-SSD-RealSense

 

By the way, sample of multistick + multithread is here.

 

https://github.com/PINTO0309/MobileNet-SSD
0 Kudos
35 Replies
idata
Employee
421 Views

"I am sorry I can not meet your expectations"

 

?

 

Pinto, just asking a question because the sticks are so fat with the heat sink I can not plug more than one into the Pi. I used a short USB to UBS cable and have had 2 plugged in but I think it blew up the Pi. It no longer starts giving me low voltage warnings. I am trying to rebuild a new one. I love your tests because you are trying some interesting combinations.
0 Kudos
idata
Employee
421 Views

@chicagobob123

 

I thought you were expecting a great suggestion on how to not use self-powered USBHub in MultiStick configuration.

 

I was misunderstanding.

 

 

just asking a question because the sticks are so fat with the heat sink I can not plug more than one into the Pi.

 

I understood what you said.

 

Certainly, Stick is enormously fat.

 

I am very happy if it is a little thinner…

 

As you say, connecting two directly to a USB port will inevitably make it impossible to connect other USB devices.

 

I love your tests because you are trying some interesting combinations.

 

Thank you very much!

 

0 Kudos
idata
Employee
421 Views

Hello.

 

I corresponded to MultiGraph and FaceDetection.

 

USB Camera + MultiGraph(PASCAL VOC + WIDER FACE) + MultiProcessing + MultiStick

 

https://github.com/PINTO0309/MobileNet-SSD-RealSense

 

https://youtu.be/fQZpuD8mWok

 

And, After changing the Good quality USB Camera and lowering the resolution, it turned out that the performance got better even with 4 Stick.

 

About 25 FPS

 

USB Camera = PlayStationEye (Second-hand goods $5)

 

https://youtu.be/v-Cei1TW88c
0 Kudos
idata
Employee
421 Views

When I increased the stick to 5, I got about 30 FPS.

 

Up to the upper limit of the bus speed of USB 2.0, performance may be improved still more.

 

I can not procure any more sticks, so I can not check it…

 

USB Camera is PlayStationEye (Second-hand goods $5)

 

Because children are asleep, I verified in the dark, and I made many mistakes in label judgment.

 

https://github.com/PINTO0309/MobileNet-SSD-RealSense

 

https://youtu.be/CL6PTNgWibI
0 Kudos
idata
Employee
421 Views

I have learned that there is a problem that the movement of the stick becomes unstable due to heat.

 

And I noticed that many engineers suffered from that problem.

 

https://ncsforum.movidius.com/discussion/1106/ncs-temperature-issue

 

To prevent thermal runaway, simple clustering function of stick was implemented.

 

However, Error handling is not implemented seriously.

 

When a certain cycle or constant temperature is reached, the active cluster switches seamlessly automatically.

 

You must turn on the clustering enable flag.

 

The default switch period is 10 seconds, the default temperature threshold is 65°C.

 

The number, cycle, and temperature of sticks constituting one cluster can be specified by the start parameter.

 

However, fine-grained control is not performed automatically.

 

Depending on your environment, please tune to the optimum parameters yourself.

 

https://github.com/PINTO0309/MobileNet-SSD-RealSense
0 Kudos
idata
Employee
421 Views

Hi, very nice work! Can you use the RealSense 435 module as source of depth information AND as RGB Camera at the same time?

 

BG, Thomas
0 Kudos
idata
Employee
421 Views

@vanthome

 

of course. :)

 

# Wait for a coherent pair of frames: depth and color frames = pipeline.wait_for_frames() depth_frame = frames.get_depth_frame()←Depth color_frame = frames.get_color_frame()←RGB
0 Kudos
idata
Employee
421 Views

nice, but this is SDK code, so it does not report itself as a normal USB webcam I suppose?

0 Kudos
idata
Employee
421 Views

@vanthome

 

Sorry, I seem to have misunderstood the meaning of your question.

 

Which of the following does your question mean?

 

     

  1. D435 + RealSenseSDK(Depth) + USB Camera(RGB) … While acquiring Depth at D435, use the USB camera at the same time.
  2.  

  3. D435(Depth) + USB Camera(RGB) … While acquiring Depth at D435, use the USB camera at the same time. However, do not use RealSense SDK.
  4.  

  5. USB Camera (Depth + RGB) … Get Depth and RGB with USB camera alone. Do not use RealSense SDK at all.
  6.  

 

If it is 1… It can be realized by modifying the program.

 

If it is 2… As long as you do not use SDK, RealSense will not work.

 

If it is 3… (1) If you know the height of the object, you can measure the coarse distance from the illuminance information with only one USB camera.

 

If it is 3… (2) If you have two USB cameras, you can calculate the distance with a simple trigonometric function.

 

Please read the article using Google Translate.

 

Distance measurement with one RGB camera (Japanese)

 

http://opencv.blog.jp/python/%E7%B0%A1%E6%98%93%E8%B7%9D%E9%9B%A2%E8%A8%88%E6%B8%AC

 

Distance measurement with two RGB cameras (Japanese)

 

https://qiita.com/ringo156/items/6c58ed5fae9e56dafefb

 

https://qiita.com/ringo156/items/9d4f81cd9aa70aa626ff
0 Kudos
idata
Employee
421 Views

Thx for your reply!

 

What I meant was 1.) I want to measure distance and use the RGB camer at the same time from the D435 so that I only need one imaging sensor and thanks, I will look at your code, I have just ordered the devices.
0 Kudos
idata
Employee
421 Views

@vanthome

 

Your attempt is very interesting.

 

However, there is one caveat.

 

The resolution of D435 can not be set lower than 640x480.

 

When using D435 and USB Camera at the same time, probably you need to have the same resolution.

 

D435 = 640x480, USB Camera = 640x480.

 

In addition, since the image expansion and contraction processing is required twice, performance may be slightly lower than the animation of my repository.

 

Since I am interested in your attempt, please tell me why you would like to use a USB camera at the same time.

0 Kudos
idata
Employee
421 Views

Hmm, maybe we have a misunderstanding. Again, I **don't **want to use a separate camera, I want to use the RGB camera built in the D435.

 

Also, as per the data sheet, the internal camera can do "RGB Resolution: 1080p @ 30fps" so why do you mention only 640x480?
0 Kudos
idata
Employee
421 Views

@vanthome

 

I seemed to be very confused.

 

And I seem to have misled you.

 

I'm sorry.

 

Once, I will organize the meaning of words.

 

◆What I meant

 

 D435 = RealSense

 

 USB Camera = Not RealSense (Ordinary USB camera)

 

◆Your meaning

 

 D435 = RealSense's Depth Camera

 

 USB Camera = RealSense's RGB Camera

 

However, D435 functions as an RGB camera without using SDK.

 

Please look below.

 

RealSense's RGB Camera + Test without using RealSense SDK

 

import cv2, time width = 1920 height = 1080 fps = "" framecount = 0 elapsedTime = 0 cap = cv2.VideoCapture(0) cap.set(cv2.CAP_PROP_FPS, 180) cap.set(cv2.CAP_PROP_FRAME_WIDTH, width) cap.set(cv2.CAP_PROP_FRAME_HEIGHT, height) cv2.namedWindow('FPS', cv2.WINDOW_AUTOSIZE) while True: t1 = time.perf_counter() ret, frame = cap.read() cv2.putText(frame, fps, (width-180,15), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (38,0,255), 1, cv2.LINE_AA) cv2.imshow('FPS', frame) if cv2.waitKey(1)&0xFF == ord('q'): break t2 = time.perf_counter() framecount += 1 if framecount >= 30: fps = "(Playback) {:.1f} FPS".format(framecount/elapsedTime) print("fps = ", str(fps)) framecount = 0 elapsedTime = 0 elapsedTime += t2-t1 cap.release() cv2.destroyAllWindows()

 

Below is FPS result. (1080p, 1920x1080)

 

I do not know if the processing performance of OpenCV can not keep up, but the performance written in the catalog did not come out.

 

xxxxx@ubuntu:~/git/MobileNet-SSD-RealSense$ python3 _usbcamera_performance_test.py fps = (Playback) 4.6 FPS fps = (Playback) 4.9 FPS fps = (Playback) 5.0 FPS : :

 

 

so why do you mention only 640x480?

 

My explanation was insufficient.

 

I cited it as an example when emphasizing performance.

 

0 Kudos
idata
Employee
421 Views

We are very disappointed that the SSD support is only for Caffee and not for TensorFlow :(

0 Kudos
idata
Employee
421 Views

@vanthome

 

I sympathize with you.

 

If I were an Intel engineer, I'm in a hurry to respond to Tensorflow. :s
0 Kudos
Reply