MATLAB RGB point cloud

[JBos3]

I would like to acquire and plot RGB colored point clouds from a D435 in MATLAB. The desired output is analogous to the rs-pointcloud.cpp example.

I reviewed rs-pointcloud.cpp and the MATLAB wrappers, and I'm able to acquire a point cloud, however I can't find how to properly color and plot the point cloud. Below is my code, which I hope will be useful to others.

There are two options that I'm considering:

1) is there a way for MATLAB to consume the realsense.pointcloud() object directly?

2) Assuming the answer to 1) is no, can I access and use the vertices and textures to reformat the RS point cloud object into a MATLAB point cloud object?

points.get_vertices(); and points.get_texture_coordinates(); seem to get correct values, but I'm not sure how to use them in MATLAB.

Thanks!

MATLAB:

% Make Pipeline object to manage streaming

pipe = realsense.pipeline();

% define point cloud object

pcl_obj = realsense.pointcloud();

% define colorizer to give point cloud color

colorizer = realsense.colorizer();

% Start streaming on an arbitrary camera with default settings

profile = pipe.start();

% Get frames. We discard the first couple to allow

% the camera time to settle

for i = 1:5

frames = pipe.wait_for_frames();

end

% Stop streaming

pipe.stop();

% Select depth frame

depth = frames.get_depth_frame();

% get point cloud points without color

points = pcl_obj.calculate(depth);

% get texture mapping

color = frames.get_color_frame();

% map point cloud to color

pcl_obj.map_to(color);

% get vertices (nx3)

vertices = points.get_vertices();

% get texture coordinates (nx2)

tex_coords = points.get_texture_coordinates();

[idata]

Hello jjbos,

Thank you for our interest in the Intel RealSense D435 camera. Please make sure that you have the latest firmware on your camera and librealsense version 2.16.1. We have modified your code (it follow the export_to_ply python examples).

function pc_example

% Make Pipeline object to manage streaming

pipe = realsense.pipeline();

% define point cloud object

pcl_obj = realsense.pointcloud();

% Start streaming on an arbitrary camera with default settings

pipe.start();

% Get frames. We discard the first couple to allow

% the camera time to settle

for i = 1:5

frames = pipe.wait_for_frames();

end

% Select depth frame

depth = frames.get_depth_frame();

% get color frame

color = frames.get_color_frame();

% map point cloud to color

pcl_obj.map_to(color);

% get point cloud points without color

pnts = pcl_obj.calculate(depth);

%export ply file

pnts.export_to_ply('pc_example.ply',color);

pipe.stop();

end

Please let us know if this works with you as well!Best regards,Eliza

[JBos3]

Hi Eliza,

Thank you. This is helpful, and *almost* what I need.

I would like to visualize the colored point cloud live stream, just like the rs-pointcloud.cpp example... but in MATLAB.

Is there a way to convert the RealSense point cloud object to something MATLAB can manipulate without having to write and then read an intermediate .ply file?

Thanks,

-Joe

[idata]

Hello jjbos,

You check out this post that I believe can help you with your project: https://www.mathworks.com/matlabcentral/answers/7202-processing-on-a-live-webcam-streaming-video https://www.mathworks.com/matlabcentral/answers/7202-processing-on-a-live-webcam-streaming-video. Please note that you might need the Image Acquisition Toolbox.

Thank you,

Eliza

[JBos3]

Hi Eliza,

Thanks for looking but this does not address the issue. The link you sent relates to processing 2D images. Please correct me if you saw something related to point clouds.

Again, I need to convert the RealSense point cloud object and other data (e.g. pc.vertices, pc.texture, color image, depth image) into a MATLAB point cloud without going through an intermediate .ply file. MATLAB requires point cloud vertices and RGB values for each vertex.

It appears I can get the correct vertices from the point cloud object but I don't know how to get the corresponding RGB values to color the point cloud. Can you help with this?

Thanks,

-Joe

Here is how I get the point cloud vertices (x,y,z values), but again I need the corresponding RGB values:

% Make Pipeline object to manage streaming

pipe = realsense.pipeline();

% define point cloud object

pcl_obj = realsense.pointcloud();

% define colorizer to give point cloud color

colorizer = realsense.colorizer();

% Start streaming on an arbitrary camera with default settings

profile = pipe.start();

% Get frames. Discard the first several to allow

% the camera time to settle

for i = 1:5

frames = pipe.wait_for_frames();

end

% Select depth frame

depth = frames.get_depth_frame();

% get texture mapping

color = frames.get_color_frame();

% map point cloud to color

pcl_obj.map_to(color);

% get point cloud points without color

points = pcl_obj.calculate(depth);

% get vertices (nx3)

vertices = points.get_vertices();

% get texture coordinates (nx2)

tex_coords = points.get_texture_coordinates();

[idata]

Hello jjbos,

The RealSense team is working on an example to demonstrate how to do this but I don't have a timeline for when it will be completed. Meanwhile, you can check out this code: https://communities.intel.com/community/tech/realsense/blog/2018/10/03/unofficial-pointcloud-stream-viewer-in-matlab https://communities.intel.com/community/tech/realsense/blog/2018/10/03/unofficial-pointcloud-stream-viewer-in-matlab.We need to figure out how to use the pcshow command and to use the same algorithm in the align.cpp example to make the depth and color frames the same size.Regards,Jesus G.Intel Customer Support

[JBos3]

Thank you. I very much look forward to the colored point cloud example in MATLAB.

-Joe

[VHess · Accepted solution]

I made this code that I thinks solve this problem:

% Make Pipeline object to manage streaming

pipe = realsense.pipeline();

% Make Colorizer object to prettify depth output

colorizer = realsense.colorizer();

% define point cloud object

pcl_obj = realsense.pointcloud();

% Start streaming with default settings

profile = pipe.start();

align_to = realsense.stream.color;

alignedFs = realsense.align(align_to);

% Get streaming device's name

dev = profile.get_device();

name = dev.get_info(realsense.camera_info.name);

% Get frames. We discard the first couple to allow

% the camera time to settle

for i = 1:5

fs = pipe.wait_for_frames();

end

%create a point cloud player

player1 = pcplayer([-1 1],[-1 1],[-1 1]);

frameCount =0;

while isOpen(player1) && frameCount < 2000

  frameCount = frameCount+1;

  fs = pipe.wait_for_frames();

  %align the depth frames to the color stream

  aligned_frames = alignedFs.process(fs);

  depth = aligned_frames.get_depth_frame();   

  color = fs.get_color_frame();

  %get the points cloud based on the aligned depth stream

  pnts = pcl_obj.calculate(depth);

  pcl_obj.map_to(color);

  colordata = color.get_data();

  colordatavector = [colordata(1:3:end)',colordata(2:3:end)',colordata(3:3:end)'];

  vertices = pnts.get_vertices(); 

  view(player1,vertices,colordatavector) 

end

% Stop streaming

pipe.stop();

[Pcmehdi]

Hello VHess,

I run your codes but I have faced an error as below:

Error using realsense.align
Error: File: align.m Line: 4 Column: 1
Illegal use of reserved keyword "classdef".
Error in realsense.VHess (line 21)
alignedFs = realsense.align(align_to);

I am using Matlab2020b!!

Do you have any idea what the issue might be?

[SVerg]

Hello! I have a problem with the code, the function

alignedFs = realsense.align (align_to);

(Error using realsense.align Error: File: align.m Line: 9 Column: 20 A constructor call to superclass realsense.filter appears after the object is used, or after a return)

Apparently there is an error in the code of align.m, could you help me with this matter? I would greatly appreciate it.

Regards,

S.V.I

[JBos3]

Thanks VHess! Works like a charm. Cheers.

[SonuMathur]

In which for you save the point cloud...?

[VHess]

@SVerg​ I had to comment out validateattributes on line 17 in align.m to get it to work, but I'm not sure if that is related to your issue.