Getting float depth data

[jb455]

Hi,

I'm trying to obtain float depth data, as the rounded/truncated integer depth data I'm currently getting is causing issues due to the 'stepping' (discontinuity when it goes from one millimetre to the next).

I remember a while ago there was an issue with the then-current SDK that QueryVertices was returning rounded/truncated integers rather than floats for depth - that is no longer the case with the 2016R2 SDK; thanks for fixing that, but:

I'm currently using the folowing code to get depth data:

var mappedImage = projection.CreateDepthImageMappedToColor(depth, , image);

mappedImage.AcquireAccess(PXCMImage.Access.ACCESS_READ, PXCMImage.PixelFormat.PIXEL_FORMAT_DEPTH_F32, out mdata);

var mwidth = mappedImage.info.width;

var mheight = mappedImage.info.height;

var mappedPixels = mdata.ToFloatArray(0, mwidth * mheight);

And then feeding those pixels to ProjectColorToCamera() to obtain the world coordinates.

The problem is that, even though I'm using the float depth format and sending it to a float array, I'm only getting integer depth values back (the values in mappedPixels are always ".0").

So: is there a bug with CreateDepthImageMappedToColor which means it only ever does integer depth, or is there something else I should be doing to get the full depth data?

I should note that I need this to be compatible with all cameras, so using the most recent SDK is not an option, nor is a fix (if this is a bug) going into a future SDK which doesn't support the R200.

Thanks,

James

[MartyG]

Getting floats instead of integers in a depth script is something that people have had trouble with in the past. A developer called x.c posted a script a about 18 months ago that they said could get float data.

Mat depthimage8;

depthimage8.create(480,640,CV_8UC1);

PXCImage::ImageData data_depth; unsigned short *depth_data;

sample->depth->AcquireAccess(PXCImage::ACCESS_READ,PXCImage::PIXEL_FORMAT_DEPTH, &data_depth);

depth_data = (unsigned short*)data_depth.planes[0];

unsigned char cVal;

for(int y=0; y

{

for(int x=0; x

{

if(depth_data[y*W+x] > fMaxValue) //fMaxValue = 1000

depth_data[y*W+x] = fMaxValue;

cVal = (unsigned char)((1.0*depth_data[y*W+x])/fMaxValue*255);

if(cVal!=0)

cVal = 255 - cVal;

depthimage8.at(y,x) = cVal;

}

}

[jb455]

Thanks for the reply Marty!

My attempt to translate that to C# has resulted in this:

var depth_data = new short[mappedPixels.Length];

var ptr = mdata2.planes[0];

var psize = Marshal.SizeOf(typeof(short));

for(int i = 0; i < mappedPixels.Length; i++)

{

depth_data[i] = (short)Marshal.PtrToStructure(ptr, typeof(short));

ptr = new IntPtr(ptr.ToInt64() + psize);

}

var dep = depth_data.Where(x => x > 0);

for (int k = 0; k < mappedPixels.Length; k++)

{

if (depth_data[k] > 1000)

depth_data[k] = 1000;

byte cval = (byte)(1.0 * depth_data[k] / (1000 * 255));

if (cval != 0)

cval = (byte)(255 - cval);

dpixels[k] = cval;

}

(where mappedPixels is an array colour.length * colour.width big)

Problem is, line 18 is never hit, so everything in dpixels is 0. depth_data fills out nicely though; dep contains lots of nonzero elements which seem reasonable, so if anyone wants to get the data out of ImageData without using ToShortArray etc for some reason, lines 1-8 will do that for them.

I don't really understand what they're trying to do in the second half of the snippet which doesn't help! (https://software.intel.com/en-us/forums/realsense/topic/557966 here is the original thread for info)

Can anyone see where I'm going wrong, or suggest any other solutions?

[MartyG]

Yes I see, because cval = 0, it does not reach line 18 because it cannot satisfy the If statement's requirement that cval does not = 0. 'For' statements are my pet hate in programming!

[jb455]

So if I use PIXEL_FORMAT_DEPTH_RAW and convert each element in the array like so:

mappedImage.AcquireAccess(PXCMImage.Access.ACCESS_READ, PXCMImage.PixelFormat.PIXEL_FORMAT_DEPTH_RAW, out mdata);

var mwidth = mappedImage.info.width;

var mheight = mappedImage.info.height;

var mp = mdata.ToShortArray(0, mwidth * mheight);

var mappedPixels = new float[mp.Length];

var du = device.QueryDepthUnit();

for (int i = 0; i < mappedPixels.Length; i++)

mappedPixels[i] = mp[i] * du / 1000;

...I still just get integers . And, actually, the values are way off too. Am I doing the conversion (line 9) right?

[MartyG]

From what I can gather, you use

PXCImage::PIXEL_FORMAT_DEPTH_F32

to get a float because that gets you a 32-bit float point (hence F32).

Intel staffer David Lu recommended to someone else with floating point troubles to look at the C# source code of the RealSense SDK sample script 'Rawstreams' at

C:\Program Files (x86)\Intel\RSSDK\framework\CSharp\DF_RawStreams.cs

[jb455]

You'd think, but as noted in my original post, that just returns integers.

I can't find anything to do with depth data in Raw Streams...

[MartyG]

Another source reckoned that "If a device fails to determine the depth of a given image pixel, a value of zero will be stored in the depth image. This is a reasonable sentinel for 'no depth' because all pixels with a depth of zero would correspond to the same physical location, the location of the imager itself."

"The default scale of an R200 device is one millimeter, allowing for a maximum expressive range of ~65 meters. The depth scale can be modified by calling rs_set_device_option(...) with RS_OPTION_R200_DEPTH_UNITS, which specifies the number of micrometers per one increment of depth. 1000 would indicate millimeter scale, 10000 would indicate centimeter scale, while 31 would roughly approximate the F200's 1/32th of a millimeter scale."

Sorry if I'm not being very helpful on this case!

[jb455]

Thanks for looking Marty, I appreciate your effort!

If I do as in post # 4 but with the standard depth image instead of the result of CreateDepthImageMappedToColor then I do get float depth values (in multiples of 1/8 of a millimetre), my problem is that it seems CreateDepthImageMappedToColor clips the data down to integers, losing the extra precision with no way to get it back.

I guess if it is bugged I'll have to use another method of mapping between the colour and depth images to get the camera coordinates. I have tried a few different methods in the past, but none of them worked as well as my current way of doing it.

[jb455]

Any comments from Intel? How did the devs intend for us to get camera coordinates with float depths, aligned to the colour camera?

[jb455]

Ok, managed to get this working using the https://software.intel.com/sites/landingpage/realsense/camera-sdk/v1.1/documentation/html/index.html?queryinvuvmap_pxcprojection.html Inverse UV Map:

PXCMImage.ImageData ddata; depth.AcquireAccess(PXCMImage.Access.ACCESS_READ, PXCMImage.PixelFormat.PIXEL_FORMAT_DEPTH_F32, out ddata);var dwidth = depth.info.width;var dheight = depth.info.height; var dPixels = ddata.ToFloatArray(0, dwidth * dheight); depth.ReleaseAccess(ddata); var invuvmap = new PXCMPointF32[color.info.width * color.info.height];projection.QueryInvUVMap(depth, invuvmap);var mappedPixels = new float[cheight * cwidth];for (int i = 0; i < invuvmap.Length; i++) { int u = (int)(invuvmap[i].x * dwidth); int v = (int)(invuvmap[i].y * dheight); if (u >= 0 && v >= 0 && u + v * dwidth < dPixels.Length) { mappedPixels[i] = dPixels[u + v * dwidth]; } }

With this, I now have float depth data aligned to the colour image, which I use in https://software.intel.com/sites/landingpage/realsense/camera-sdk/v1.1/documentation/html/index.html?projectcolortocamera_pxcprojection.html ProjectColorToCamera to get the camera coordinates for each point in the colour image.

[MartyG]

Thanks for sharing, JB!

[jb455]

Grr, just tried this with the R200 (above was all testing with an SR300) and I'm getting ints again! @intel: is it possible to get float depth data from an R200 or is that int only?

[idata]

Hi James,

Let us investigate if this is possible. We'll get back to you as soon as we have some updates on your case.Did you try using the exact same snippet you used for the SR300? We just want to make sure you didn't change anything in your code.Regards,-Pablo

[jb455]

Yes, just plugged the R200 in using the same code. Thanks!

[idata]

Hi James,

I'm not sure if you've checked the RF_MeasurementSP.exe sample, it uses floats to measure objects using the R200 camera. You can check the source code in the following directory C:\Program Files (x86)\Intel\RSSDK\sample\RF_MeasurementSP\RF_MeasurementSP_vs2015.sln, hopefully you can implement a similar solution for your current project. Regards,-Pablo

[jb455]

Hi Pablo, thanks for the reply.

I hadn't seen that sample before. Looking through it now (and it's quite hard to tell what's going on as it has very few comments), it seems as though they get the z values from QueryVertices (or the ScenePerception equivalent). However, when I run QueryVertices on the R200 the returned z values are ints (the same with the SR300 gives float z values). So: is there something I'm missing in that sample? Could you give line numbers which do what I need please? Or is it just the case that ScenePerception does things differently to the normal camera streaming and can get higher precision?

Thanks!

[idata]

Hi James,

Let me take a closer look at the source code myself, just to see what's happening within it and provide a more accurate answer. Thank you for your patience.Regards,-Pablo

[idata]

Hi James,

I believe this is what you're looking for: Measurement.cpp (line 30) -> float Measurement::GetDistance(const unsigned int uiIndex1, const unsigned int uiIndex2) constYou can check references to this function to see how it is used.Regards,-Pablo

[idata]

Hi jb455, could you please tell me how would this code be in c++, because i´m getting a lot of errors when i try to write this same code in c++.

Thank you!!

[jb455]

Migueel: The Projections sample does similar stuff: look for the QueryInvUVMap method in projection.cpp. That should help you modify my code.