pixels mismatch

[fwang22]

When I deal with the color images and depth images, I find the pixels are not consistence in the color and depth image. ,there is an offset between the color images and the corresponding depth images. So what should I do?

[MartyG]

I apologize for any confusion. In summary:

• If you are aligning 2D images and recording it as a bag, and then trying to convert the bag file into a 3D point cloud ply: that would not be the best approach, because of needing to take camera extrinsics into account to get accurate results when doing alignment in a 3D point cloud.

• If though you are happy to just do the color and depth alignment in the 3D point cloud mode of the RealSense Viewer and export a ply from that mode, that should be fine (3D mode exports ply files instead of bag files, so you do not need the rs-convert tool).

[fwang22]

What you mean is that the color and depth alignment in 3D of RealSense Viewer is not in the bag file, this pattern exports only one .ply file.

So is the point cloud exported from .bag file calculated in depth image?

To sum up the question I have always wanted to ask: how do I align the color and depth images I have obtained in matlab? Thank you .I'm sorry to trouble you by asking you this question all the time.

[MartyG]

No need to apologize. I understand how frustrating this process must be for you.

A bag file recorded using the RealSense Viewer captures whatever streams and settings are active at the time. So for example, if you do not have an IR stream active when recording then the IR stream is not included in the bag file. By that logic, if color and depth are not aligned in 2D mode in the RealSense Viewer when recording the bag, the streams should be unaligned in the bag file when it is played back.

As far as I am aware, the bag file recording (2D mode) and the ply file recording (3D mode) are completely separate. When recording a point cloud in 3D mode, it creates a ply file directly and does not do a conversion of bag to ply).

If you are using MATLAB, it may be easier to do the alignment using a MATLAB 'function' (a program you download to extend the abilities of MATLAB). At the link below, you can find a couple of example programs for aligning depth with color in MATLAB.

https://uk.mathworks.com/matlabcentral/fileexchange?term=align+depth+color

There is also a downloadable function that can convert a depth image into a point cloud within MATLAB. Scroll down the page to the heading titled 'Depth Image to Point Cloud (MATLAB)' and click on the 'depthToCloud.m' link to launch its download in your browser.

https://rgbd-dataset.cs.washington.edu/software.html

[fwang22]

I want to know how to align color and depth in the RealSense viewer.

[tpain]

yes, this is a fact that the RealSense Viewer can align color with depth in its 3D point cloud mode due to this pixel mismatch causes. https://epsonsupports.net/blog/how-to-fix-printer-offline/ helped me to know about this case. Thanks for sharing this information with us.

[MartyG]

fwang22, please ignore the tpain post above about 'epson support', it is advertising 'spam'.

As far as I know, there is not a built-in way to align color and depth in 2D mode in the RealSense viewer (only in 3D mode), and a 2D align has to be done with scripting.

It is possible to bring the two closer together by using the Viewer's post-processing 'Decimation Filter' , but this is a crude way to do it that is not as good as true alignment in my opinion.

https://forums.intel.com/s/question/0D50P0000490XQOSA2/in-the-intel-realsense-viewer-both-the-depth-image-window-and-the-rgb-image-window-are-not-mismatched-like-the-pixel-position-is-different-in-the-two-images-how-can-i-get-the-same-pixel-position-in-the-two-images-for-d435#

[Sahira_Intel]

Hi @fwang22​ 

To "align color and depth" in the RealSense Viewer, you would follow the instructions Marty G. gave above and switch to 3D mode in the Viewer, and change the texture source to color. You can then export this .ply file.

You can also align depth frames to their corresponding color frames by using the rsalign sample (this is a c++ program) here: https://github.com/IntelRealSense/librealsense/tree/master/examples/align

I see that you'd like to try to do this in MatLab so (like Marty G. said above) there is a program written in MatLab for that called align.m here: https://github.com/IntelRealSense/librealsense/blob/master/wrappers/matlab/align.m

If i am missing something, please let me know!

Regards,

Sahira

As always @MartyG​ , thank you for your help!