Saving and Retrieving Depth Data

[jb455]

Hi,

I'm building an application where I take colour & depth images and allow the user to do things requiring the depth points and vertices at a future time.
I've gotten it working by:

1. Take image (colour & depth)
2. Get depth points by ImageData.ToUShortArray
3. Map these points to the colour image (or vice-versa) using Projection.MapColourToDepth
4. Get the vertex array by Projection.QueryVertices

When the user does things straight after the image is captured it's no problem because I can just map points as we need them - the user may only be interested in 5 points so it runs quickly.
My problem is, to allow the user to bring back the image later and use the depth data for different points, the way I've done it is to map ALL of the colour points (640*480 image = 307,200 points), save to a text file then read it back later. Unfortunately, the mapping when you have so many points takes over a minute which doesn't fit with the flow of the application.

Does anyone know of a better way to save depth data to be rebuilt later? (In a way that I can get vertices & project to colour too)

Thanks!
James

[Xusheng_L_Intel]

There are lots samples in RealSense SDK to save depth and color image and replay them in the future. Of course, you also can use those data in the future. You can find those sample codes @C:\Program Files (x86)\Intel\RSSDK\sample\DF_RawStreams

[jb455]

Hi David, thanks for your reply.

I've checked out the RawStreams sample and it only seems to record video - is there a way to save just a single frame?
I've also now found and tried the PXCMPhoto.SaveXMP method (which is called SaveXDM in the online docs weirdly) but get a "PXCM_STATUS_HANDLE_INVALID" when I try to run it. Is there something special I need to do to the PXCMSession to enable this feature (apart from Session.CreatePhoto())? Also, is this available for use on both cameras? The only example of it I can find is in the R200 Enhanced Photography C++ sample.

Thanks again!

Edit:
Just noticed the C++ sample uses PXCMSenseManager.QuerySession() to update the session before doing session.CreatePhoto(), so I've put this in now but still get the same error.
 

Edit 2:
Right, just discovered that you have to fill in the PXCMPhoto via PXCMPhoto.ImportFromPreviewSample (I'd assumed that when you did CreatePhoto it would initialise it with the last frame captured), but it's still not working. The docs say that the Sample photo must include a Projection element, but mine only contains colour and depth elements (ir, left and right are null). Is there something I need to do to set up a projection and attach it to the Sample?

Edit the third:
Realised that, though I'd updated the SDK on my machine, I hadn't replaced the .dlls in my solution with the new ones, hence the SaveXDM method having a different name. Alas, this doesn't solve my problem :(

[jb455]

Has anyone managed to get the SaveXDM function working?

Here's the current state of my code:

Start of routine:

session = PXCMSession.CreateInstance();
pSenseManager = PXCMSenseManager.CreateInstance();

In while loop:

PXCMCapture.Sample photo = pSenseManager.QuerySample();
session = pSenseManager.QuerySession();
depthPhoto = session.CreatePhoto();
depthPhoto.ImportFromPreviewSample(photo);

Then later when I want to save:

string fileName = "D:\test.jpg";
pxcmStatus status = depthPhoto.SaveXDM(fileName);
if (status < pxcmStatus.PXCM_STATUS_NO_ERROR)
     logger.Error("Problem saving file " + fileName + ": " + status);

Unfortunately every time I run it status is STATUS_HANDLE_INVALID which doesn't really tell me anything about what to do to fix it! So, has anyone been successful with using this? Any ideas where I'm going wrong?

Thanks!
James
 

[jb455]

Ugh. I just thought "hey, what if..." and yep. The slash in the filename was the wrong way around. Please consider putting in more descriptive error codes!

[jb455]

Ok so my next problem is when it comes to loading up the image later. I can load the image as a PXCMImage and get the ImageDate etc but I also need to create a projection in order to map depth points to the colour image and to retrieve the vertices. Unfortunately when I do device.CreateProjection() it comes out as null. The docs say that function will return null "if the interface is not available". Does anyone know how to make the projection available here? Or if it is possible to do so from a loaded depth image? My code is below.

            PXCMSession session = PXCMSession.CreateInstance();
            PXCMPhoto photo = session.CreatePhoto();
            pxcmStatus status = photo.LoadXDM("D:/test.jpg");

            if (status < pxcmStatus.PXCM_STATUS_NO_ERROR)
                logger.Error("problem loading depth image:" + status.ToString());
            PXCMImage depth = photo.QueryDepthImage();

            PXCMImage.ImageData ddata;
            depth.AcquireAccess(PXCMImage.Access.ACCESS_READ, out ddata);
            UInt16[] dpixels = ddata.ToUShortArray(0, depth.info.width * depth.info.height);

            PXCMSenseManager pSenseManager = PXCMSenseManager.CreateInstance();
            pSenseManager.Init();
            var device = pSenseManager.captureManager.QueryDevice();
            PXCMProjection projection = device.CreateProjection();
            PXCMPoint3DF32[] vertices = new PXCMPoint3DF32[depth.info.width * depth.info.height];

            projection.QueryVertices(depth, vertices);

This works great until the last line (dpixels is populated and correct, for example), where I get "Object reference not set to an instance of an object." because projection is null.

James

[Michael_M_9]

I've run into the same problem with the Java implementation of projection.queryVertices(...)

The only significant difference with my code is that I created the senseManager from the factory method on PXCMSession:

PXCMSenseManager senseMgr = session.CreateSenseManager();

James, did you find a solution?

Thanks,

Michael

[jb455]

Sadly no. In the end this was taking too much time and I wasn't able to get anywhere so I moved on to other things, awaiting some updates from Intel on this matter. I will have to go back to it some point though so if anyone has a fix or any advice I'd be very grateful!

I remember one of the Intel guys saying in another thread that the two SenseManager creation methods were the same, one just a shortcut of the other, so I don't think that'll make a difference, but I've made a note to test that the next time my code compiles (it's in pieces at the moment!) Thanks!