convert depth image to point cloud

[fwang22]

How to convert depth frame to point cloud using Matlab? I know one way is if we know the camera parameters,we can obtain point cloud .But what are the parameters ?or Is there any other way to do it in Matlab？

[idata]

Hello Nevyn,

The Matlab Wrapper in librealsense is just an interface to the core librealsense functionality. The librealsense libraries are implemented in C++. The Matlab wrapper for librealsense contains two examples, depth_example.m and rosbag_example.m, that are standalone and perform a task. These examples are there to show you how to build your own Matlab code using the wrapper files. The rest of the Matlab wrapper files, for example pointcloud.m, syncer.m, pipeline.m, etc, are just interfaces to librealsense classes that are implemented in C++. Pointcloud.m gives you access to the class rs2::pointcloud.

If you want to generate a pointcloud using Matlab, you can start by looking at the rs-pointcloud.cpp (notice it is C++) example found in librealsense/examples/pointcloud then code the same flow in Matlab. There is no pre-built rs-pointcloud example in Matlab, you will have to create it yourself. In your Matlab pointcloud code, you will have to call the functions in pointcloud.m. You will see that pointcloud.m has functions for calculate and map_to. These two functions also exist in the rs2:pointcloud class in C++.

Also, regarding your issue with rs-convert. Please make sure that the when recording the .bag file, the Depth stream and the RGB stream are recorded at the same resolution and the same FPS in order to get a very close number of depth and color images. I hope this information helps you! Best regards,Eliza

[idata]

Hello Nevyn,

Could you please let us know if the information provided above was useful? Thank you,Eliza

[fwang22]

Hello Eliza,

Thank you the information provided,it is useful for me.

I have obtained the point cloud by rs-convert,and obtained the color images through matlab ,but I found the number of point cloud by rs-convert is not consistent with the number of depth image by matlab.

The .bag files need to be decompressed before we obtain the color images in matlab. I don't know if it's because of decompression or if rs-convert is missing some frames.

Thank you,

Nevyn

[idata]

Hello Nevyn,

Thank you for the confirmation.The problem you are encountering is caused by rs-convert. This is a known issue and the Engineers are currently working to resolve it. It will be solved on the next librealsense release. Thank you,Eliza

[fwang22]

Hello Eliza,

Thank you for the information.

I want to know what is the essential difference between these two cameras on imaging.

Thank you !

Nevyn

[fwang22]

Hello Eliza,

when I obtain the .bag file through viewer,I can obtain the RGB and depth frames are not synchronous, I know the depth camera and the color camera work independently,but how to deal with it?

Thank you ,

Nevyn

[idata]

Hello Nevyn,

When you record the .bag file, do you use the same resolution and FPS? Could you please explain what do you mean by "I can obtain the RGB and depth frames are not synchronous" ? Where do you notice it? In RealSense Viewer or after you make the conversion?Also, could you please specify what is the camera that you are using? D415 or D435? Thank you,Eliza

[fwang22]

Hello Eliza,

I record the .bag files using the same resolution and FPS.

I get the .bag files, but when i read the .bag file through matlab， I found that the number of the color frame is not consistent with the number of the depth frame.

I get the point clouds by the re-convert.exe, and the number of the point cloud is also inconsistent with the color frames or depth frames bu matlab。

Thank you,

Nevyn

[idata]

Hello Nevyn,

The rs-convert has a known issue that it drops frames. Because of this reason, it will not output the same number of frames as in the bag file. It will be solved with the next librealsense release. Regarding the matlab issue, could you please provide us the code you are using to read the number of frame?Thank you,Eliza

[fwang22]

Hello Eliza,

I use the MATLAB codes to read the depth freams from .bag files,the codes as follows:

bag = rosbag('file.bag'); % Load the rosbag.

rosbag info 'file.bag'; 

bSel = select(bag, 'Topic', '/device_0/sensor_0/Infrared_1/image/data'); % Select a specific topic from the above information. It should end with /image/data if you want the frames.

msgs = readMessages(bSel); % Read all the messages (here, frames) in that topic as an nx1 cell array.

% Iterate through the cell array and read each frame.

for i = 1 : numel(msgs) img = readImage(msgs{i}); % Do something with the frame (img).end

and then I convert the depth frames to point clouds,

But i have another question that are the depth camera and the color camera in synchronous? Because the depth frames is not consistent with color frames when I set the two cameras same resolution and FPS.

Thank you!

Nevyn

[fwang22]

Hello Eliza,

I want to know if there are code commands available for both depth camera and color camera synchronization?

Thank you,

Nevyn

[idata]

Hello Nevyn,

Thank you for the information provided. The RealSense engineers are still working on mapping color on Matlab examples. You can check /community/tech/realsense/blog/2018/10/03/unofficial-pointcloud-stream-viewer-in-matlab this post. For the alignment between Depth and RGB frames, you can check out the https://github.com/IntelRealSense/librealsense/tree/master/examples/align align example provided in RealSense SDK 2.0 if you want to start developing it yourself. Thank you,Eliza

[fwang22]

Hello Eliza,

If I can set two cameras align frames through other ways? For example,parametres of codes or in viewer.

When I obtain the color frames and depth frames from .bag files through MATLAB，I can see the frames number of depth is not consistent with frames number of color，such as 66 depth frames and 65 color frames，is not 66 depth frames and 66 color frames.

Thank you.

Nevyn

[idata]

Hello Nevyn,

If you want to align the depth and RGB color though other ways than the one provided by RealSense SDK, you will have to develop your own program that does that. As for the color and depth frame number that does not match. Could you check out the rs-rosbag-inspector Tool from the SDK and let us know if the frame number matches there?Thank you,Eliza

[fwang22]

Hello Eliza,

If there are MATLAB codes or C codes about converting the depth frame to point cloud?

Thank you

Nevy

[MartyG]

You can convert depth to point cloud directly in MATLAB with a downloadable function. On the link below, scroll down the page to the heading titled 'Depth Image to Point Cloud (MATLAB)' and click on the 'depthToCloud.m' link to launch its download in your browser.

https://rgbd-dataset.cs.washington.edu/software.html RGB-D (Kinect) Object Dataset

[fwang22]

Hello ELiza，

I made a comparison for the frames was extracted from .bag files through rs-convert.exe and MATLAB rosbag codes ways, and I found that it's not same.

the MATLAB result as follow:

the rs-convert.exe result as follow:

we can see the depth frames and color frames of the same .bag file are different.

Dose this mean there is a missing frame in rs-convert.exe?Is it true in MATLAB?

Thank you ,

Nevyn

[MartyG]

Loss of frames in a bag have been a known problem. Rosbag fixes were added in RealSense SDK version 2.16.1. Are you using this version, please?

[fwang22]

Yes， I using the version 2.16.1,so has the loss frames question been solved?

[MartyG]

This issue was last mentioned 9 days ago on the RealSense GitHub, when a member of the RealSense development team asked for an update from a community member who asked about the rs-convert problem about whether using 2.16.1 had solved the problem. That user has not responded yet.

https://github.com/IntelRealSense/librealsense/issues/1919# issuecomment-427262310 problems in rs_convert · Issue # 1919 · IntelRealSense/librealsense · GitHub

A user did offer their own solution for rosbag reading though, which can be downloaded via the zip link below. It is a trustworthy link as the program has been added to the SDK's directory of community-contributed samples.

https://github.com/IntelRealSense/librealsense/files/2272586/RealSenseHelper.zip https://github.com/IntelRealSense/librealsense/files/2272586/RealSenseHelper.zip

Information about the program can be found here:

https://github.com/IntelRealSense/librealsense/issues/2215 rosbag reader · Issue # 2215 · IntelRealSense/librealsense · GitHub

[fwang22]

For 2.16.1 version, I tested a lot of group data,and found that individual data was inconsistent,such as,for a same .bag file,

or the number of point clouds are the same as the depth frames,but not the same as the color frames.

I will try what you say.

Thank you