How to find the closest point of an object in front of the camera?

[Somesh_N_]

As of now, I have my ImageData data structure populated. How can I find the depth values and correspondingly the closest point to the camera and its depth value  in the image? I'm having difficulty understanding planes and pitches, especially due to the lack of examples in C#.

Kindly help me in this regard.

Thanks!

[Andradige_S_Intel]

Planes and pitches: are you referring to image data? that is a common industry standard. In very high level, a format like NV12 will have two planes, one for Y and the other for the inter-leaved UV. Pitch is the stride one had to make in reading the planes, normally in most RealSense formats, that is just the width of the frame. This was about the color frame, you get the depth image in one plane.

To get the closest point in depth, you can try something like PXCBlobExtractor::BlobData structure which is in the form of

struct BlobData {

  PXCPointI32  closestPoint;

  PXCPointI32  leftPoint;

  PXCPointI32  rightPoint;

  PXCPointI32  topPoint;

  PXCPointI32  bottomPoint;

  PXCPointF32  centerPoint;

   pxcI32       pixelCount;

};

Refer to the manual for more information

[Jonathan_M_Intel]

Hi, there's an easy way to get to the raw depth data, but it's non-obvious so I understand your confusion.

To get to the depth map, you must use code like this (pseudocode):

unsigned short depthValues[width * height] = {};

while (loop) {

sensemanager->AcquireFrame();

PXCCapture::Sample * sample = sensemanager->QuerySample();

PXCImage * depthMap = sample->depth;

PXCImage::ImageData data;

depthMap->AcquireAccess(PXCImage::ACCESS_READ, &data);

memcpy(depthValues,data.planes[0],sizeof(unsigned short) * width * height);

depthMap->ReleaseAccess(&data);

sensemanager->ReleaseFrame();

}

In most cases, it is not useful to worry about pitch values, and the edge cases are too complex to describe here when the above will be useful for most cases.

Once you have the data in your own array, you can usually loop through it looking for the lowest non-zero z value.  The data will be in "raster order", or row-by-row, so once you find your minimum value, its position in the array will tell you what its original (x,y) position was.  Then you can create a new PXCPoint3DF32 and use the PXCProjection interface to convert that single point to world coordinates.  Something like this:

int minX, minY;

unsigned short minZ = 0xFFFF;

for(int i = 0; i < height; ++i) {

 for(int j = 0; j < width; ++j) {

  if(depthValues[i * width + j] < minZ) {

      if(depthValues[i * width + j] == 0) continue;

      minZ = depthValues[i * width + j];

      minX = j;

      minY = i;

   }

 }

}

PXCPoint3DF32 originalPoint, worldOutputPoint;

originalPoint.x = minX;

originalPoint.y = minY;

originalPoint.z = minZ;

pxcprojection->ProjectColorToCamera(1, &originalPoint, &worldOutputPoint);

Hope this helps!

[Somesh_N_]

Thank you, PUBUDU S. and Jonathan T. for taking time to  help me out. So far I've been using BlobData structure to find the closest point, however my program needs the actual z depth values themselves which is unavailable within the structure.

I'm sure the snippets shared by Jonathan T. would be very helpful. I had come barely close to this explanation in the online documentation, but I must admit, I wasn't quite able to understand.The explanation here is quite clear.  I'll proceed in this direction, get the code to work and try to resolve this thread.

Thank you again!

[Dagan_E_Intel]

Hi Somesh,

have a look at the mask_utils sample.

look for the next line:

            convertTo8bpp((unsigned short*)data.planes[0],info.width*info.height, charBuffer);

Here we take the depth image and convert it from 2 bytes stream to single byte stream.
It will show you how to iterate depth image pixels.
In your case you don't really need to iterate it, once you have the closest point coordinates, and the depth buffer ((unsigned short*)data.planes[0]) you can get the desired pixel directly