D435 USB-powered depth camera

[JRomu]

I am interested D435 for an application we developed with Kinect.

1. Can the above said camera work in a completely dark room? 2. Will depth information get effected by turning on or off the light in the room? – Will the sudden change of light effect depth data at that time 3. What are kind frames available for extraction? 4. Can I extract IR frame ?

Will be using C++ . IDE is MS Visual Studio 2015.

Your early reply on the above will be much appreciated. Thanks Johan

[MartyG]

1. RealSense IR sensors require a certain amount of light in order to generate a good quality image. If the location is too dark then the image will be mostly or completely black.

I ran a test though with a mostly dark room at 7 am (not completely dark) with the lights off and blinds shut with a D415 camera, and still got a decent IR image.

2. When the lights were turned on, it was a very smooth, instant transition to the stronger lighting conditions.

3. You can extract RGB color, depth and IR. There are different formats for each of these main stream types. The link below has a script with a list of image formats on lines 54 to 74

https://github.com/IntelRealSense/librealsense/blob/5e73f7bb906a3cbec8ae43e888f182cc56c18692/include/librealsense2/h/rs_sensor.h# L55 librealsense/rs_sensor.h at 5e73f7bb906a3cbec8ae43e888f182cc56c18692 · IntelRealSense/librealsense · GitHub

4. If you use the RealSense Viewer tool that comes with SDK 2.0 to generate the infrared stream, you can snap a single frame using the SDK 's record and playback functions. To capture a frame instead of recording a sequence of frames with the Record button, click on the Pause icon to pause the stream, and then click on the Snapshot button next to it to capture that paused frame as a PNG image.

[JRomu]

Hi, Thank you for your reply.

I will be using OpenCV libraries and C++ to develop my own application. I would like to know is there a C++ sample code to extract IR frame and depth data.

Thanks once again.

Johan

[MartyG]

The quickest method would likely to be to take the Capture sample (which shows color and depth streams) and alter the code to show the IR stream instead of the color one.

https://github.com/IntelRealSense/librealsense/tree/master/examples/capture librealsense/examples/capture at master · IntelRealSense/librealsense · GitHub

[JRomu]

Hi,

Thank you very much that was very helpful.

Just to give you an idea of the application I am planning to develop;

https://www.youtube.com/watch?v=YaPshferTeI&t=23s Mavis - YouTube

I did that with Kinect and Kinect is no more and trying to find a alternative.

Further what is the recommended specs for a PC to use D435?

Best Regards

Johan

[MartyG]

That's a very cool application!

The D415 and D435 cameras work with any Intel or ARM processor and only need a USB 3.0 port.

[JRomu]

Thanks

[JRomu]

HI,

Finally, I received my D435 after a long await.

At this time I am using windows 10 to develop my prototype application but eventually move to Ubuntu.

I installed the latest SDK 2.12.0. I am using MS Visual studio 2015 for compiling the code.

Following code compiles without any problem

quote

// License: Apache 2.0. See LICENSE file in root directory.

// Copyright(c) 2017 Intel Corporation. All Rights Reserved.

# include // Include RealSense Cross Platform API

# include // Include OpenCV API

int main(int argc, char * argv[]) try

{ // Declare depth colorizer for pretty visualization of depth data rs2::colorizer color_map;

// Declare RealSense pipeline, encapsulating the actual device and sensors

rs2::pipeline pipe; // Start streaming with default recommended configuration pipe.start();

using namespace cv;

const auto window_name = "Display Image"; namedWindow(window_name, WINDOW_AUTOSIZE);

while (waitKey(1) < 0 && cvGetWindowHandle(window_name))

{ rs2::frameset data = pipe.wait_for_frames(); // Wait for next set of frames from the camera rs2::frame depth = color_map(data.get_depth_frame());

// Query frame size (width and height)

const int w = depth.as().get_width(); const int h = depth.as().get_height();

// Create OpenCV matrix of size (w,h) from the colorized depth data

Mat image(Size(w, h), CV_8UC3, (void*)depth.get_data(), Mat::AUTO_STEP);

// Update the window with new data

imshow(window_name, image); }

return EXIT_SUCCESS;

}catch (const rs2::error & e){ std::cerr << "RealSense error calling " << e.get_failed_function() << "(" << e.get_failed_args() << "):\n " << e.what() << std::endl;<p>  return EXIT_FAILURE;}catch (const std::exception& e){ std::cerr << e.what() << std::endl;<p>  return EXIT_FAILURE;}

Unquote

But when I run the application I am getting following error;

API version mismatch: librealsense.so was compiled with API version 2.9.1 but the application was compiled with 2.12.0! Make sure correct version of the library is installed (make install)

Please advice.

Thanks

Johan

[JRomu]

HI,

Please ignore my question. I worked it out. Coping the realsense2.dll file to application directory fixed the problem.

I would like to how can get non colorizer raw depth frame gray scale and also the depth information of x,y coordinates.

In OpenNi2 I was able to save depth info from frame to Unit_16 array. Is it possible to do the same?

Thanks

Johan

[MartyG]

I researched your question carefully but there are no precedents to refer to for a Windows user - the small number of cases involving this error are all based on the Linux version of the SDK.

The solutions in those Linux cases could be summed up as:

1. Compile the 2.12.0 SDK from source code instead of using a pre-made executable, if the pre-built SDK is what you are using.

Or

2. Download the old 2.9.1 version of the SDK and try your program with that.

https://github.com/IntelRealSense/librealsense/releases/tag/v2.9.1 Release Intel® RealSense™ SDK 2.0 (build 2.9.1) · IntelRealSense/librealsense · GitHub

[JRomu]

Hi,

Thanks.

But I fixed the problem by copying the realsense2.dll to the root directory of the application.

Any idea how can get the depth data to array for me to process it, ie to find depth at a certain x,y.

Thanks

Johan

[MartyG]

Awesome news that you found a solution to your problem!

The discussion linked to below may be helpful to you.

[JRomu]

Hi,

Thanks for that link. It was very helpful. I have some more work to do but so far progress well. What I have achieved so far;

I will keep you posted.

Thank you once again.

[MartyG]

Looks good - thanks for sharing your progress.

[JRomu]

Hi,

I progressed further, I was able to get the depth data in meters for each frame and process it.

I have posted the sample code how to do it in previous link you provided, so it may be useful for some one.

I have one challenge to overcome. I am processing the IR frame and it seems to get effected by turning on the light in the room. Kinect IR frame does not get effected by it.

Any advice on this is welcomed.

Thanks

Johan

[MartyG]

I did some tests a few months ago with turning the room light on and off to see how the camera was affected in the moment of change between dark and light.

You can use scripting to turn off a component called the IR Emitter that assists the IR Sensor with exposure but can cause saturation in the IR sensor if the lighting is too dark or too bright.