How Do I... Get Depth Data per Pixel in Unity3D

Lance_R_ · ‎05-06-2015

Here's what I found from the user community.

https://software.intel.com/en-us/forums/topic/537482

This C++ method was posted in the forum thread I linked above to store the depth information of the image per pixel:

///////////////////////////////////////////////////////////
PXCImage::ImageData depthData;
if(sample->depth->AcquireAccess(PXCImage::ACCESS_READ, PXCImage::PIXEL_FORMAT_DEPTH, &depthData) >= PXC_STATUS_NO_ERROR)
{
    // depth image height and width: 640 x 480
    unsigned short depthValues[640*480];
    memcpy(depthValues,depthData.planes[0], sizeof(unsigned short)*640*480);
    // Necessary to release the access:
    sample->depth->ReleaseAccess(&depthData);
}

// use the depthValues array
////////////////////////////////////////////////////////////

And I totally understand that method the way it is. Coming up with that method on my own at this point would be a bit of a challenge. Regardless, I'm glad there are some examples people have shared for storing depth information per pixel from the camera. What I'm interested in however is accessing this information from a Unity script.

One of my concerns while figuring this out is that it says in the SDK:

"The application must use the provided member functions to access the image buffers." If you want to take a look at those member functions in the PXCImage class, the link that in the recent documentation is here:

https://software.intel.com/sites/landingpage/realsense/camera-sdk/2014gold/documentation/html/index.html?manuals_member_functions9.html

Anyway my main question for now basically is, how would I call upon and use this information in Unity3D as a Unity script? Is there a tutorial or link available on how to access lower level class objects and structs and use them in Unity3D?

Once I get this "depthData" I want to set new color values based upon how far away from the camera each pixel point is in reality.

Thanks in advance for any help anyone may have.

MartyG · ‎05-06-2015

I suppose what you could do is compile the above C++ script into a DLL file and put it into the Plugins.managed folder of your project's Assets panel.

This Unity doc page says how to compile C++ into a DLL and use it in your Unity project.

http://docs.unity3d.com/Manual/UsingDLL.html

Lance_R_ · ‎05-06-2015

Marty, although I haven't tried it yet, just from my general understanding of the C# scripting in Unity I think that would be an unnecessary step. If the PXCMImage namespace is accessible in C# via the RSSDK there should be a way to easily include these libraries at the beginning of any C# Unity script.

The above code example uses C++ syntax, yes. And I do not know for sure whether C++ libraries can be included in a C# Unity script. Compiling a new DLL in Visual Studio would just make the code more "difficult to work with" in Unity.

I should just be able to say something like the following:

using UnityEngine;
using System.Collections;
// using [whatever the namespace is for the location of the C# RSSDK library, here?];

public class SomeClassOfMine /* : PXCMSenseManager  */ // [?]
{

     void Start() {
     
     }

     void Update() {
          // algorithm in C# syntax to do what I need here, etc.
     }
}

MartyG · ‎05-06-2015

It's my understanding that C++ and C# are very different these days, having drifted apart over time, so converting C++ to C# isn't so easy.

Since you're using Unity, I wonder if using a script like that is over-complicating the problem. If I was approaching it myself, I think I might consider using a 'TrackingAction' or 'TranslateAction' SDK script and track the Z-position value of a particular object, since Z would change as the hand or face moved towards the camera or away from it.

Lance_R_ · ‎05-07-2015

Marty,

That isn't what I want to do at all. You're right though. C++ libraries are not C# libraries. You can totally program in ANSI C in either language however. Like, for example, printf and scanf are just instrinsic parts of the C language core. But that isn't my point. That's totally off topic. What I'm saying is the "algorithm" is perfectly viable. It would be great if I could just say, like...

#include <cstdlib>

using namespace std;

...or whatever right at the top of a Unity script, but I don't think that's possible. That's basically what I meant.

PXCMImage exists in the C# library. The RSSDK dynamic link library "libpxcclr.unity.dll" already contains this.

All I have to do is rewrite the syntax. That's what I was thinking, if I didn't say it up there. But I'm very new to this SDK. I would think that you'd have to call, something like...

using UnityEngine;
using System.Collections;
using System.Collections.Generic;
// using RSUnityToolkit;       // <---- unnecessary 
     // "libpxcclr.unity" in the plugins.managed folder exposes the RSSDK
     // I kind of thought for auto complete in Monodevelop to find PXCMImage... 
     // you'd need to declare that you're "using" something.  That isn't the case.

public class MyClassName : BaseAction  // it would inherit from the BaseAction it seems?
{
          void Update() {
                    // algorithm in C# syntax to store pixel depth data per update in an array
                    // and render them out, probably using a Sense AR game object
                    // as re-colored pixels based on the depth
          }

 }

Let me make a little more clear what I'm going to do. I'm not tracking an object. I'm re-coloring the pixels based on how far away every pixel's Z-position is from the camera. So...

if (the Z-position is greater than 3 feet away)
colorOfThatPixel = blue;

MartyG · ‎05-07-2015

I apologize, Lance. C++ code writing is outside of my area of experience. It's definitely something I'm interested in learning when the next RealSense app contest comes around. I'd have no trouble suggesting a way to recolor if it was objects you wanted to recolor rather than pixels.

Don't know if this will be any use, but there's a RealSense documentation page that mentions a C# function called PXCMProjection.QueryUVMap that maps depth to color.

https://software.intel.com/sites/landingpage/realsense/camera-sdk/v1.1/documentation/html/index.html?queryuvmap_pxcprojection.html

MartyG · ‎05-07-2015

I found this C# script in the RealSense documentation, which maps depth to color coordinates using UV mapping.

// Create the PXCMProjection instance.
PXCMProjection projection=device.CreateProjection();
 
// color and depth image size.
PXCMImage.ImageInfo dinfo=depth.info;
PXCMImage.ImageInfo cinfo=color.info;
 
// Calculate the UV map.
PXCMPointF32[] uvmap=new PXCPointF32[dinfo.width*dinfo.height];
projection.QueryUVMap(depth, uvmap);
 
// Translate depth points PXCPointF32[] uv to color PXCPointF32[] ij
for (int i=0;i<uv.Length;i++) {
   ij.x=uvmap[(int)uv.y*dinfo.width+(int)uv.x].x*cinfo.width;
   ij.y=uvmap[(int)uv.y*dinfo.width+(int)uv.x].y*cinfo.height;
}
 
// Clean up
projection.Dispose();

Lance_R_ · ‎05-07-2015

Thanks Marty, I'll have to read that carefully.

On a side note, I noticed you've linked to the v1.1 version of the documentation. The most recent version of the SDK is the 2014GOLD. Right off the bat when I began searching online for information, I began to realize that there are multiple (and deceptively similar) iterations of the SDK reference guide. The "Perceptual Computing" reference guide is out of date and definitely not the one to use. Between v1.1 and 2014GOLD, I'm not sure what the differences are in their entirety. However, the most correct link at the present time would be here:

https://software.intel.com/sites/landingpage/realsense/camera-sdk/2014gold/documentation/html/index.html?queryuvmap_pxcprojection.html

No worries. Thanks for your help thus far!