Mapping RGB points to Depth using PXCProjection

[Ravish_C_]

I am trying to map color points to depth points, and vice versa. To do this, I create instance of PXCProjection from device, and multiply each point with the transformation value in uvmap.

My code is based on the sample on the API docs. When I run this, I get a null value for the PXCProjection variable, which should be a valid interface because the session has been initialized, and both the depth and color streams are enabled.

Here, I am calling the method translateDepthToColor by passing in defined projection, and ImageInfo arguments. points is a vector of struct Point, which has integer values x and y.

void translateDepthToColor(PXCProjection *projection, PXCImage::ImageInfo dInfo, PXCImage::ImageInfo cInfo) {
 PXCPointF32 *uvmap = new PXCPointF32[dInfo.width*dInfo.height];
 projection->QueryUVMap(depthImage, uvmap);

 for (int i = 0; i < points.size(); i++) {
  points.x = uvmap[(int)points.y * dInfo.width + (int)points.x].x * cInfo.width;
  points.y = uvmap[(int)points.y * dInfo.width + (int)points.x].x * cInfo.height;
 }

 delete[] uvmap;
 return;
}

void translatePoints(Feed type)
 PXCProjection *projection = device->CreateProjection();
 

 PXCImage::ImageInfo dInfo = depthImage->QueryInfo();
 PXCImage::ImageInfo cInfo = colorImage->QueryInfo();

 if (type == COLOR1) {
  translateDepthToColor(projection, dInfo, cInfo);
 }
 else if (type == DEPTH1) {
  translateColorToDepth(projection, dInfo, cInfo);
 }
}

I see two possible approaches, one using uvmap, and the other my calling the corresponding mapColorToDepth and mapDepthToColor methods. I haven't tried the latter methods, so am unsure if they are a better fit for my program. I do not see why projection would return a null value, other than if session was not initialized, which it is

 //in main
        session = PXCSenseManager::CreateInstance();
 session->Init();
 device = session->QueryCaptureManager()->QueryDevice();
 std::cout << "device initiated " << device->CUID << std::endl;
 PXCVideoModule::DataDesc streams = {};
 if (session->QueryCaptureManager()->QueryCapture()) {
  session->QueryCaptureManager()->QueryCapture()->QueryDeviceInfo(0, &streams.deviceInfo);
 }
 else {
  streams.deviceInfo.streams = PXCCapture::STREAM_TYPE_COLOR | PXCCapture::STREAM_TYPE_DEPTH;
 }

 session->EnableStreams(&streams);

Both depth and color streams are rendered properly, which means that session was able to query both streams. Is there another part of the camera initialization process I am missing here, or an error I am making in querying the uvmap?

[Xusheng_L_Intel]

Please provided more detail code so I can help you. Where did you define the device and where did you call translatePoints? You can find our sample codes how to use projection and uvmap C:\Program Files (x86)\Intel\RSSDK\sample\DF_RawStreams. Thanks!

[jb455]

Hi Ravish,

I've been trying to get this working myself but haven't had much luck. Where have you defined depthImage?

Side point, and not much use if you can't get the projection working, but I noticed though on line 7 of your first block of code, I think [...].x * cInfo.height; should be [...].y * cInfo.height;.

Also, I've used projection.MapColorToDepth (and vice-versa) quite a lot and found them useful for mapping a small number of points at a time, but if you want to map a large section of the image it takes a long time to process (>1 minute for 640*480 image).

My problem is with the InvUVMap in C#. Every point in the map comes out as either (-1,-1) or (0,0), so when I loop through and multiply the coordinates by depth.width or depth.height I end up with an array of points which are either (-640,-480) or (0,0).

private PXCMPoint3DF32[] MapAllColourPointsToDepthByUV()
        {
            PXCMPoint3DF32[] dp = new PXCMPoint3DF32[cwidth * cheight];

            PXCMPointF32[] invUVMap = new PXCMPointF32[cwidth * cheight];
            pxcmStatus st = projection.QueryInvUVMap(depth, invUVMap);
            for (int y = 0, k = 0; y < cheight; y++)
            {
                for (int x = 0; x < cwidth; x++, k++)
                {
                    dp = UVtoDepthPoint(invUVMap[y*cwidth + x]);
                }
            }
            return dp;
        }
        private PXCMPoint3DF32 UVtoDepthPoint(PXCMPointF32 uv)
        {
            try
            {
                int u = (int)uv.x * dwidth;
                int v = (int)uv.y * dheight;
                int z;
                if (u > 0 && v > 0)
                    z = dpixels[(int)(v * dwidth + u)];
                else 
                    z = -1;
                return new PXCMPoint3DF32(u,v,z);
            }
            catch(Exception e)
            {
                logger.Error(e.ToString());
            }
            return new PXCMPoint3DF32();
        }

I should note that parallel to this I'm using QueryVertices and MapColorToDepth which are working as expected so I don't think there's anything wrong with the projection unless you have to do something special to use the UVMap.

Has anyone gotten this working with C#? Then only SDK example I can find is C++.