topic Re: D435 Resolution VS Depth Image Quality in Items with no label

D435 Resolution VS Depth Image Quality

FDing3 — Tue, 14 Aug 2018 18:19:31 GMT

Hi,

I recently tested the depth quality using Intel Depth Quality Tool with printed textured paper. I found the Subpixel RMS Error increases when I changed the resolution from 848x480 to 1280x720. According to the equation in Best-Known-Methods to tune up D400 camera PDF(the equation is included in the attachment), the depth RMS error decrease as the resolution increases. I also saw the PDF says the higher resolution will have negative impact on the performance of the application too. I am not clear about what kind of negative impact though.

Since I found the skeleton tracking module provided by Nuitrack works much better under the 1280x720 resolution for D435 instead of 848x480, I need to know the impact of the higher resolution on the depth quality.

Also, could you take a look at two depth quality test pics in attachments to see if the camera is well calibrated?

My last question is I calculated the limit of Depth RMS error at the distance of 1111mm. It's 4.6578mm for Xres=848 and 3.0858 for Xres=1280. What's use of limits we get here? Are there any relationship between the Depth RMS error and Plane Fit RMS error?

Thanks!

Re: D435 Resolution VS Depth Image Quality

idata — Thu, 16 Aug 2018 09:35:22 GMT

Hello jakeding,

Thank you for your interest in the Intel RealSense D435 camera.

We have acknowledged your request and will provide a resolution a soon as possible.

Thank you,

Eliza

Re: D435 Resolution VS Depth Image Quality

idata — Thu, 16 Aug 2018 14:59:05 GMT

Hello jakeding,

I will treat every question one by one.

I need to know the impact of the higher resolution on the depth quality

The higher the input resolution, the better the input image, the better the depth precision.

The only compelling reason to set the D435 cameras into a lower resolution mode would be to reduce the minimum operating range, which will scale with the x-y resolution and to reduce the bandwidth on the USB3.0 bus, to allow, for example, more cameras to operate at the same time, if the bandwidth limit has been reached.

This information is also shown at page 1 from Best-Known-Methods for Tuning Intel® RealSense™ D400 Depth Cameras for Best Performance paper.

Could you take a look at two depth quality test pics in attachments to see if the camera is well calibrated?

Yes, the camera seem to be well calibrated.

What's use of limits we get here? Are there any relationship between the Depth RMS error and Plane Fit RMS error?

Depth RMS error can be quantified in either distance (e.g., mm) or disparity (e.g., pixels) units, where the Plane Fit RMS Error calculates the RMS Error of each valid pixel from the best fit plane in mm and Subpixel RMS error calculates the RMS Error of each valid pixel from the best fit plane in pixels.

This information can also be found on Depth Testing Methodology paper at pages 9 and 10.

Please let me know if I can provide you any other information!

Kindest regards,

Eliza

Re: D435 Resolution VS Depth Image Quality

FDing3 — Thu, 16 Aug 2018 17:21:20 GMT

Dear Eliza,

I do appreciate your professional answers here!! Actually, I posted my questions after I read all white papers provided by Intel. I am confused because I found there are some contradictions between my experiment results and the corresponding theories in Intel papers. Let me clarify all my questions in the following way.

I need to know the impact of the higher resolution on the depth quality

The reason why I ask this: As you can see in the depth quality snapshots in the original attachments, both the Subpixel RMS error and Plane-fit RMS error are actually bigger under 1280x720 resolution than those under 848x480 resolution with the same camera configurations and same physical location.

Like the plane fit RMS is 3.32mm for 1280x720 and 2.71mm for 848x480 and the Subpixel RMS is 0.09 for 1280x720 and 0.05 for 848x480. It's obviously against with the saying that the higher the input resolution, the better the input image, the better the depth precision. I got bigger noise in higher resolution, why is it?

Are there any relationship between the Depth RMS error and Plane Fit RMS error?

The reason why I ask this: You successfully clarified part of my question here but not the whole picture. Now I know the unit and usage of Plane Fit RMS error and Subpixel RMS error.

I calculated the Depth RMS error from my experiment as mentioned in the original post, and it seems has nothing to do with the Plane Fit RMS error and Subpixel RMS error limit. So how do I get the theoretical limit of (or relate to) Plane Fit RMS error and Subpixel RMS error from the theoretical limit of Depth RMS error equation?

Thanks!!

Re: D435 Resolution VS Depth Image Quality

idata — Mon, 20 Aug 2018 16:58:13 GMT

Hello jakeding,

We have forwarded your questions to the RealSense engineering team. We will update this thread as soon as we have a response from them.

Regards,

Jesus G.

Intel Customer Support

Re: D435 Resolution VS Depth Image Quality

idata — Wed, 22 Aug 2018 08:40:43 GMT

Hello jakeding,

Thank you for your patience.

The Depth RMS error formula in the BKMs for Tuning RealSense whitepaper shows how to translate between the Plane Fit RMS Error and the Subpixel RMS Error. The formula on Page 3 of the BKMs for Tuning whitepaper can be interpreted as:

Depth RMS error (mm) = Plane Fit RMS error = Distance(mm)^2 x Subpixel RMS Error/(focal length * Baseline)

The reason the Plane Fit RMS error increases with resolution on the D435 camera is because the active pixels (see page 9 of the BKMs for tuning whitepaper and page 33 of the D400 datasheet) on the D435 are 1280 x 800. When the camera returns depth, it uses rectified images. During the rectification process, the image gets downsized. In order to get the 1280x720 resolution after rectification, the extra pixels are extrapolated. This is why the whitepapers say that the optimal depth resolution for the D435 is 848x480. Using a resolution higher than 848x480 should give you similar or slightly worse depth quality.

The D415 camera's depth sensors have active pixels of 1920x1080, which is higher than the D435, so it's optimal depth is 1280x720 (as can be seen on page 1 of the BKMs for Tuning whitepaper).

The subpixel RMS error goes up with resolution because the pixels get smaller and there are more pixels, therefore the disparity between them gets larger.I hope this information helps you!

Best regards,

Eliza

Re: D435 Resolution VS Depth Image Quality

FDing3 — Thu, 23 Aug 2018 20:29:20 GMT

Hi Eliza,

Thank you very much!!!

My last question about his topic is: What the meaning or usage of the theoretical error limit calculated from Depth RMS error (mm) = Plane Fit RMS error = Distance(mm)^2 x Subpixel RMS Error/(focal length * Baseline)?

Like I got 2.88mm theoretical limitation of Depth RMS error(Plane fit error) calculated using the above equationusing D435 with HFOV=90deg, Xres=848, baseline=50mm, subpixel=0.08 and Distance = 1074.44mm (1074.44mm is provided by Realsense Viewer and you can see it in original attachment. I don't know what's the definition of the Distance in the equation. Should I use the ground truth value measured by myself instead of the distance shown in Viewer?) while the Realsense Viewer gives me 3.32mm(Plane fit error).

And I got 4.35mm for HFOV=90deg, Xres=848, baseline=50mm, subpixel=0.08 and Distance = 1073.62mm while Realsense Viewer gives me 2.71mm(Plane fit error).

How do I understand the relationship between the theoretical Depth RMS error limit and the Depth RMS error provided by the Realsense Viewer? And please tell me should I use the ground truth value to put into the above equation?

Thanks for your time and effort!

Jake

Re: D435 Resolution VS Depth Image Quality

idata — Fri, 24 Aug 2018 13:45:47 GMT

Hello jakeding,

When calculating the theoretical limits of the Depth RMS error will be better to use the ground truth value.

I have also calculated the Depth RMS Error using the values from the screenshots of the Depth Quality Tool and this is what I got:

(1280x720) HFOV = 90deg, Xres = 1280, subpixel = 0.09, baseline = 50mm, distance = 1111mm. The Plane Fit RMS error is: 3.47mm (The Depth Quality Tool indicated 3.32mm)
(848x480) HFOV = 90deg, Xres = 848, subpixels = 0.05, baseline = 50mm, distance = 1111mm. The Plane Fit RMS error is: 2.91mm (The Depth Quality Tool indicated 2.71mm)

As you can see, the Xres value will change with the resolution. Also, you have different values for subpixels for 1280x720 and for 848x480.

The difference between the values received from the theoretical calculus and the values provided by the Depth Quality Tool are caused by the fact that every camera has a slightly difference focal length, field of view, etc.

Kindest regards,

Eliza

Re: D435 Resolution VS Depth Image Quality

FDing3 — Fri, 24 Aug 2018 13:59:16 GMT

You are awesome! Thanks again!

Re: D435 Resolution VS Depth Image Quality

FDing3 — Fri, 24 Aug 2018 16:14:42 GMT

Hi Eliza,

Last two questions:

Why do you use Subpixel error instead of Subpixel in the equation? In P3 of BKMs white paper, I found they recommend us to use Subpixel = 0.08.

Also, in my understanding, the Depth error limit should be the best case scenario. This means the theoretical limit should always be smaller than the realistic error computed from Realsense Viewer. How do you think the definition of the depth error limit?

Best,

Jake

Re: D435 Resolution VS Depth Image Quality

idata — Fri, 24 Aug 2018 16:57:50 GMT

Hello jakeding,

Subpixel RMS error and Subpixel represents the same value on the BKMs paper. This value was set as 0.08 in order to provide with a better understanding of the relationship between Depth RMS Error and the distance for D415 and D435 cameras. (the graph on page 4)

Regarding your last question, I do agree with you. The theoretical limit should be the best scenario, but as I have mentioned before, all the cameras are slightly different due to manufacturing differences. In the https://www.intel.com/content/dam/support/us/en/documents/emerging-technologies/intel-realsense-technology/Intel-RealSense-D400-Series-Datasheet.pdf datasheet, you can also see that there are tolerances for FOV, at page 56. When we calculate the theoretical limit of the Depth RMS error, we are using some specific values, which may or may not be the same as yours. For example, if your camera has a smaller HFOV value than 90deg, then your Depth RMS error from the Depth Quality Tool will be smaller as well.

I hope this answers your questions!

Thank you in advance,

Eliza

Re: D435 Resolution VS Depth Image Quality

FDing3 — Thu, 30 Aug 2018 14:59:45 GMT

Hi Eliza,

Sorry for the late reply. Now I know the Subpixel RMS error and Subpixel represents are the same value.

But I am still confused about theoretical depth RMS error limit equation on BKMs paper. My question is: Why is this equation a limit? I think we just input the real physical parameters like focal length or distance and get the error. So, what's the definition of the limit here? If the limit here indicates the best scenario, why did I have the smaller RMS error (3.32 & 2.71) generated from Depth Quality tool than the theoretical RMS error (3.47 & 2.91)?

I found a paper which derive the theoretical depth RMS error equation(please see attachment) derived from depth equation. I don't know if you derive in the same way.

Best,

Jake

Re: D435 Resolution VS Depth Image Quality

idata — Thu, 30 Aug 2018 17:21:06 GMT

Hello

Please note that the refer of "limit" appears because it is a mathematical equation that does not take into account all physical imperfections in the actual hardware. The equation will come very close to the reported values by the tool but the sensor hardware may introduce tiny variations.

Please let me know if you have further questions!

Best regards,

Eliza

Re: D435 Resolution VS Depth Image Quality

FDing3 — Thu, 30 Aug 2018 17:26:02 GMT

Thanks! So can I understand the imperfections in the camera may cause the realistic error less than the theoretical limit value?

Re: D435 Resolution VS Depth Image Quality

idata — Thu, 30 Aug 2018 17:49:50 GMT

Hello jakeding,

Yes, this is correct.

Thank you,

Eliza

Re: D435 Resolution VS Depth Image Quality

FDing3 — Thu, 30 Aug 2018 17:58:34 GMT

Thank you.