pixels mismatch

[fwang22]

When I deal with the color images and depth images, I find the pixels are not consistence in the color and depth image. ,there is an offset between the color images and the corresponding depth images. So what should I do?

[MartyG]

The physical RGB and IR imager sensors in the camera are in different positions. So I would recommend performing an Align operation.

In the link below is a tutorial for aligning color with depth.

https://github.com/IntelRealSense/librealsense/tree/master/examples/align

[fwang22]

I want to know if there is a way to align an image in the viewer, or how to align a color image with its corresponding depth image use code because I have obtained the color and depth images

[MartyG]

An alternative to using the Align example program is to refer to this guide on the Align processing block:

https://github.com/IntelRealSense/librealsense/wiki/Projection-in-RealSense-SDK-2.0?language=en_US#frame-alignment

The RealSense Viewer can align color with depth in its 3D point cloud mode. I do not know of a way to do alignment in ithe Viewer's default 2D mode though. A point cloud can have color and depth aligned in 3D mode using the process below:

STEP ONE

Enable both the depth stream and the RGB stream whilst in the default 2D mode

STEP TWO

Click on the '3D' button in the top corner of the RealSense Viewer to enter 3D point cloud mode.

STEP THREE

Click on the drop-down menu called 'Texture Source' and select the 'Color' option. Do not change the 'Depth Source' menu above it.

RGB and depth should now be aligned in the point cloud.

[fwang22]

I've tried this before. The point cloud extracted from the.bag file is color - and depth-aligned, but in a two-dimensional image the color - and depth-aligned image is not。Is that true?

[MartyG]

Yes, it is usually recommended that 2D alignments are done with code, using scripting such as the Align processing block that I linked to in my previous message.

An alternate method to the rs-align sample program to align color and depth is in the sample program 'grabcuts'

https://github.com/IntelRealSense/librealsense/tree/master/wrappers/opencv/grabcuts?language=en_US

[fwang22]

I wonder if alignment can only be done when sampling, there is no way to align the samples that have been collected.

If select the 'Color' option in the'Texture Source' , point cloud and color are pixel aligned?

[MartyG]

It is possible to do processing on non real-time data that has already been recorded, such as .raw and .bag files or PNG images of color and depth.

Here are a couple of example discussions about doing so with images.

https://github.com/IntelRealSense/librealsense/issues/2231

https://github.com/IntelRealSense/librealsense/issues/2534

[fwang22]

I want to do it use matlab code, because the above discussions is so confusing for me.

[MartyG]

If you have a Windows PC, there is a compatibility 'wrapper' for MATLAB in the RealSense SDK 2.0 that can do alignment with a function called 'align.m'.

https://github.com/IntelRealSense/librealsense/tree/master/wrappers/matlab

The discussion linked to below talks about using align.m to do alignment between color and depth images.

https://github.com/IntelRealSense/librealsense/issues/2455

[fwang22]

I have seen the answer :

https://github.com/IntelRealSense/librealsense/issues/2455#issuecomment-439939852

but I think it aligns the images in real time.

[fwang22]

I tried this method to align color images and depth images, but the results were disappointing.

Another problem, when I extract the point cloud using convert, there are always errors in CMAKE.

I have tried some methods, but it still doesn't work. Can you tell me how to solve it? Thank you.

[MartyG]

Another way to do alignment may be with a MATLAB product called the Image Processing Toolbox. It is not inexpensive, but it does have a free trial version.

https://uk.mathworks.com/products/image.html

[fwang22]

I don't know if the above method works for aligning color and depth images,but I will try it . Thank you very much for your timely reply, which is very important to me.

[fwang22]

I tried this method to align color images and depth images, but the results were disappointing.

Another problem, when I extract the point cloud using convert, there are always errors in CMAKE.

I have tried some methods, but it still doesn't work. Can you tell me how to solve it? Thank you.

[MartyG]

I tracked down a further discussion on aligning non-live depth and color images.

https://github.com/IntelRealSense/librealsense/issues/1274

[fwang22]

I want to use ‘rs-convert’ to get the point cloud, because I used to extract depth also for point clouds. But when I extract the point cloud, it's always terminated.

[MartyG]

The rs-convert tool is for converting bag files to other formats, such ans PNG (image) and ply (point cloud). Are you trying to convert a bag file, please?

[fwang22]

Yes, I want to convert .bag file into .ply file,but the 'rs-convert' always stops as above.

[MartyG]

If you need the point cloud to have color and depth aligned, doing a bag to ply conversion of 2D data is probably not the best way to do it. This is because 3D point clouds need to take the "extrinsic" values of the camera into account when doing alignment, otherwise the results will be inaccurate.

I will refer this case directly to an Intel support member in the hope that they can provide a quick answer.

 @Jesus G.​ 

[fwang22]

In fact, what I need is the color images and the point cloud. I used depth images to get the point cloud before, but found that the point cloud was not aligned with the color images in this way.

I asked you this question before. You said click on the '3D' button in the top corner of the RealSense Viewer to enter 3D point cloud mode. And click on the drop-down menu called 'Texture Source' and select the 'Color' option. Do not change the 'Depth Source' menu above it.

I did the same when collecting data. So I thin this should be color aligned with the point cloud.