How can I extract the point cloud and RGB, XYZ arrays from "bag" file with Matlab, when I record a scene using RealSense Viewer in Win10? I am using RealSense D415&D435 and the data are stored in "bag" file.

[VT000]

I tried with matlab functions: rosbag, select, readMessages, but I can't extract the point cloud and the XYZ array of the scene/frame. I only got the RGB or the INFRA image data tih mentioned functions (the dimensions should be 640x480x3). Can RealSense Viewer save other file types, not only "bag" and "ply"?

[MartyG]

Hi there! Are you the user named 'TDVL' who asked a similar question on the GitHub recently, please?

https://github.com/IntelRealSense/librealsense/issues/3241

The GitHub discussion is likely to be the best place to get help for most of your questions, especially since that conversation is so far progressed already. Regarding your file type question though, the SDK does have a utility called 'Convert' that can convert a bag file to a range of other formats.

The documentation and source code of this tool is available from the link below.

https://github.com/IntelRealSense/librealsense/tree/master/tools/convert

There is also a pre-built executable-file version of Convert in the SDK folder location Intel RealSense SDK 2.0 > Tools

[VT000]

Hello MartyG,

Yes, I tried to use the same username, but I can't do it somehow.

Thank you for the answer about file types, it is OK now.

But, how can I extract the point cloud and the RGB, XYZ arrays from the stored "bag" file with Matlab? I tried with point_cloud example and rosbag_example from Matlab wrapper, but the results are not good.

Also, with Matlab functions from Robotic Toolbox I got the RGB and infra data, but I didn't get the XYZ and the point cloud.

How can I do this? Naturally, I there are few frames, the RGB and XYZ arrays should be stored in cell. LAter, these data I can store in *.mat file.

FIrst, how can I read it from "bag" file using Matlab? Also, there is no manual or tutorial for the RealSense Matlab wrapper.

Can you help me with this. I need this data for image processing. The RealSense is used only for recording.

Thank you in advance.

[MartyG]

There is not much information about getting a point cloud from a bag with MATLAB. From your references to rosbag_example and the robotics toolbox, it sounds like you have already read the best discussion on the forum about this subject.

https://forums.intel.com/s/question/0D50P0000490WSjSAM/matlab-how-to-get-points-from-each-fram-of-a-bag-file?language=en_US

You might be able to bypass using the RealSense SDK and do a depth to point cloud conversion in MATLAB directly. This can be done with a downloadable 'function' extension

On the page linked to below, there is a MATLAB function that can be downloaded that converts a depth image to a point cloud in MATLAB. Scroll down the page to the heading titled 'Depth Image to Point Cloud (MATLAB)' and click on the 'depthToCloud.m' link to launch its download in your browser.

https://rgbd-dataset.cs.washington.edu/software.html?language=en_US

[VT000]

Dear Marty,

I will download the Matlab function and try it. If I see correctly, I need to save the scene in *.png format? Than how can I get the correct RGB and XYZ arrays of the scene? In need them to detect obstacles.

But, still I don't know what is wrong with Matlab wrapper? The "pointcloud_example" function from the SDK gives me only one XYZ array, on the other hand the RealSense Viewer records a larger number of frames (I should get as much arrays as I have frames, or not)? Maybe I use it wrong, I don't know. How can I get XYZ vertices for all frames separately with pointcloud_example function?

Maybe you can write me an example for using the Matlab wrapper files? There is no tutorial for the wrapper and some solutions are not obvious (I am not the only person with these problems, you can see the forum).

So, I need a pointcloud1 message in Matlab with RGB and XYZ arrays (my friend got this with ROS, but I am not a ROS user and I don't need it for image processing). Is it possible to get from bag file somehow?

If you provide me your email, I can send you one *.bag file recorded with the Viewer and one *.mat file which contains the point cloud (pointcloud1 message, I can view it with pcshow in Matlab, this is what I need). Maybe you will see something I do not? Also, I can send you the changed pointcloud_example with my modifications to call the data from bag file.

Thank you for comment and for the help.

Also, thank you in advance.

[MartyG]

You have a point about the MATLAB function and its use of PNGs. It may not have been intended for users who need to get the XYZ coordinates in addition to generating the point cloud.

RealSense programming is not one of my expert areas unfortunately, so I can only base my answers on what I already know or can research from available information. This is why the GitHub discussion will be a better place to get advice on programming. I understand the problems caused by having to sometimes wait for replies though.

If you are reading a bag file, the number of frames in the file does not always match up to what you would expect it to theoretically be, and recordings may have less frames than the ideal maximum. Numerous people have experienced this when playing back bag files. As far as I know there isn't a method to guarantee that every frame will be present in the recording, with no losses.

I wonder if a viable strategy might be to use the point cloud example program in the RealSense SDK MATLAB wrapper and edit it to use a bag file as the source for the data instead of live streaming.

https://github.com/IntelRealSense/librealsense/blob/master/wrappers/matlab/pointcloud_example.m

If you look at the first lines of the rosbag example program in the MATLAB wrapper, you can see instructions for telling the script to use a file as the data source instead of the camera.

https://github.com/IntelRealSense/librealsense/blob/master/wrappers/matlab/rosbag_example.m#L3

[VT000]

Dear Marty,

RealSense programming is not my area also, I only need the point cloud and the images captured by RealSense.

Hence, how should I use the original pointcloud_example function? Plug the camera in and just run the pointcloud_example function? What is the output of the function? I can't find any manual for this.

How can I store the data from live stream as point cloud from RealSense using the Matlab wrapper?

I hope you can guide me.

Thanks.

[MartyG]

I do not work for Intel and have the same amount of access to documentation as the public does. So if there is not much documentation for a feature, then I cannot provide information unless somebody else - Intel or a member of the RealSense community - writes it, unfortunately.

An alternate approach to your problem may be to export a ply file from the RealSense Viewer's 3D mode instead of a bag and then load it into MATLAB with the pcread instruction, displaying the point cloud in MATLAB.

https://uk.mathworks.com/help/vision/ref/pcread.html

Here is a YouTube video too about loading a ply in.

https://www.youtube.com/watch?v=njIB2CeBV84

[VT000]

Dear Marty,

I can store it in PLY format, but the problem are the dimensions are not preserved. I need a 640x480x3, namely 307200x3 arrays and the PLY format gave me less. Then, I can not in the smaller dimensions to specific dimensions. The result will not be correct.

Do you know the way of live stream usement of pointcloud_example function? How will store this function the point cloud?

Thank you in advance.

[MartyG]

You may certainly be able to save a .raw image in the RealSense Viewer's 2D mode and then import it into MATLAB with scripting such the the code in the link below.

https://forums.intel.com/s/question/0D70P0000068bdzSAA

However, my research could not find any way to convert a .raw file to a point cloud in MATALAB. Every effort to solve the problem gets blocked at this challenge.

[VT000]

I forgot earlier. I have exported the depth image in *.png and I have tried the depthToCloud.m file. I got an error, but I used it like in instructions:

Array dimensions must match for binary array op.

Error in depthToCloud (line 31)

pcloud(:,:,1) = xgrid.*depth/constant/MM_PER_M;

I don't know why is this.

[VT000]

Dear Marty,

I tested the pointcloud_example function and I got a set of parameters and variables as result. One of them is pointcloud 1x1, but I can not open that variable to see the content. I assume that this variable should contain the RGB and XYZ arrays, like when I read in a PLY file.

What can be the problem?

Here is the code also:

pipe = realsense.pipeline();

   pointcloud = realsense.pointcloud();

   % Start streaming on an arbitrary camera with default settings

 %profile = pipe.start();

   cfg = realsense.config();

   cfg.enable_device_from_file('D:\Work\Measurements\+realsense\new_h_a1.bag');

   profile = pipe.start(cfg);

   %figure('visible','on'); hold on;

   %figure('units','normalized','outerposition',[0 0 1 1])

   % Main loop

   for i = 1:10000

       % Obtain frames from a streaming device

       fs = pipe.wait_for_frames();

       % Select depth frame

       depth = fs.get_depth_frame();

       color = fs.get_color_frame();

       % Produce pointcloud

       if (depth.logical())% && color.logical())

           pointcloud.map_to(color);

           points = pointcloud.calculate(depth);

           % Adjust frame CS to matlab CS

           vertices = points.get_vertices();

           X = vertices(:,1,1);

           Y = vertices(:,2,1);

           Z = vertices(:,3,1);

          % plot3(X,Z,-Y,'.');

          % grid on

          % hold off;

          % view([45 30]);

          % xlim([-0.5 0.5])

          % ylim([0.3 1])

          % zlim([-0.5 0.5])

          % xlabel('X');

           %ylabel('Z');

          % zlabel('Y');

           pause(0.01);

       end

        %pcshow(vertices); Toolbox required

   end

    % Stop streaming

   pipe.stop();

Thank you in advance.

[VT000]

Hello Marty,

I changed the pointcloud_example function from RealSense Matlab wrapper to pull the stream from the bag file. I have got some variables and objects, but I can't access to 'poincloud' object.

I am sending you some images to see the Matlab command window and the Help Document for the object 'pointcloud'. You can see, that there are few variables-objects and there are two pointcloud type variables.

Also, when I export the scene as a ply file, the resolution is always lower then 640x480x3, it is 320x240 I think. I need this 640x480x3 resolution (307200x3 XYZ and RGB arrays). Is this possible? Can I set the resolution for exporting the ply file?

Finally, I sent one bag file captured with the Viewer to my friend to and he can't open it with ROS. There was an error. So, even the ROS can't open the bag file to give back me the point cloud like pointcloud1 message to open in Matlab later.

One customer suggested me an OpenNI. What do you think about that?

[VT000]

Hello Marty,

I saved the *.png files and other metada, but the resolution is 320x240. I need 640x480, at least. The importing of the *.raw file with fopen in not a problem, but I still don't have the pointcloud1.

I have run the pointcloud_example function (without change), but I didn't get anything as output, only two empty figures? Where are stored the data? Hence, I tried to record with Matlab function, without the Viewer, using only default camera settings (this is not my goal, I need the special setups in the Viewer) for start, but I didn't get outputs.

What is wrong with this function? Maybe you will see something to change.

I can send you the bag files and one correct *.mat file if you send me your email. Maybe that will help you (and me after you see it).

[Eliza_D_Intel]

Hello VT100, My colleague Alexandra have provided useful information in regards to ply files and why 640x480 can't be achieved. - https://forums.intel.com/s/question/0D50P00004F2O1G/can-i-change-the-resolution-to-640x480x3-when-i-export-a-scene-to-ply-file-with-realsense-viewer-in-win10 Could you please describe here what are the problems you encounter in Matlab, now having this information? Thank you and best regards, Eliza

[VT000]

Dear Eliza,

Thank you for the answer. Regarding the PLY file, that is only a spare idea. This is OK now.

But, the main idea is to get the point cloud with corresponding XYZ values and RGB values in 640x480x3 (307200x3 arrays) form the "bag" file captured by RealSense Viewer in Matlab. How can I do this?

When I use the pointcloud_example function set to pull the stream from bag file, I can't access the pointcloud object. I sent the screenshot.

I have opened a ticket about my problem already, MartyG was the main commentator on Intel's forum. This issue is opened on GitHub also. No solutions yet. MartyG recommended me to contact the Intel Suppert Team.

If I can't do this from the "bag" file, I hope I can do this using live stream with Matlab wrapper, but how? What is the syntax for this in Matlab? I tried using pointcloud_example function, but I didn't get anything. Also, can I use the Viewer's parameter setups (edge preserving filter, hole filling etc.) during live streaming?

Kind regards

[JBos3 · Accepted solution]

Hello,

I can't help you with the "bag" file. I haven't used this feature yet. But, I can help you acquire a point cloud directly from the RealSense camera. See the code below:

Good luck!

-Joe

1) Here is a link to code for acquiring an RGB colored point cloud in MATLAB (read comments for a quick fix to an issue related to align.m superclass):

https://forums.intel.com/s/question/0D50P0000490XiJSAU/matlab-rgb-point-cloud

2) The following code retrieves the XYZ values of the point cloud (no color)

% Make Pipeline object to manage streaming

pipe = realsense.pipeline();

% define point cloud object

pcl_obj = realsense.pointcloud();

% Start streaming on an arbitrary camera with default settings

pipe.start();

% Get frames. We discard the first couple to allow

% the camera time to settle

for i = 1:5

frames = pipe.wait_for_frames();

end

% get pointcloud from RS

% define point cloud object

pcl_obj = realsense.pointcloud();

% get depth frame, used to calculate point cloud  

depthFrame = frames.get_depth_frame();

% generate pc and texture mappings

pnts = pcl_obj.calculate(depthFrame);

vertices = pnts.get_vertices();

% optional: populate MATLAB point cloud object

pCloud = pointCloud(vertices);

% display point cloud

pcshow(pCloud);