RealSense + EmguCV

[Pol_P_]

I am trying to smooth a depth image acquired with RealSense and then project it to world coordinates using PXCMProjection. My code is in c# and I am trying to use EmguCV library to perform the smoothing operations on the image. I keep running in formatting errors.

Here is my code:

public static PXCMPoint3DF32[] toDepthData(PXCMImage image, PXCMProjection proj)
{
    PXCMImage.ImageData data;
    image.AcquireAccess(PXCMImage.Access.ACCESS_READ, out data);

    PXCMPoint3DF32[] pts= new PXCMPoint3DF32[image.info.width * image.info.height];
    proj.QueryVertices(image, worldPts);

    image.ReleaseAccess(data);
    image.Dispose();

    return pts;
}

public PXCMPoint3DF32[] smoothDepthData(PXCMSession session, PXCMImage image, PXCMProjection proj)
{
    // get image info
    PXCMImage.ImageInfo info = image.info;
    // get image data
    PXCMImage.ImageData data;
    image.AcquireAccess(PXCMImage.Access.ACCESS_READ, PXCMImage.PixelFormat.PIXEL_FORMAT_DEPTH, out data);

    // create a bitmap from the image data
    Bitmap dBmp = data.ToBitmap(0, info.width, info.height);

    // release access to original image and dispose
    image.ReleaseAccess(data);
    image.Dispose();

    // create a opencv image
    Image<Gray, byte> dImg = new Image<Gray, byte>(dBmp);

    // performing some OpenCV smoothing operations here
    // ...

    // convert opencv image to bitmap
    dBmp = dImg.Bitmap;
                
    // create new PXCMImage with the same dimensions as the original
    PXCMImage smooth = session.CreateImage(info);

    // get access to the data of the new image
    PXCMImage.ImageData sData;
    smooth.AcquireAccess(PXCMImage.Access.ACCESS_WRITE, PXCMImage.PixelFormat.PIXEL_FORMAT_DEPTH, out sData);

    // copy bitmap data
    BitmapData smoothData = new BitmapData();
    smoothData.Scan0 = sData.planes[0];
    smoothData.Stride = sData.pitches[0];
    smoothData.PixelFormat = dBmp.PixelFormat;
    smoothData.Width = dBmp.Width;
    smoothData.Height = dBmp.Height;
    BitmapData finalData = dBmp.LockBits(new Rectangle(0, 0, dBmp.Width, dBmp.Height), ImageLockMode.ReadOnly | ImageLockMode.UserInputBuffer, dBmp.PixelFormat, smoothData);

    // relese image data
    smooth.ReleaseAccess(sData);
    dBmp.UnlockBits(finalData);
   
    return toDepthData(smooth, proj);

    smooth.Dispose();
}

My problem is that I only can transform PXCMImage instances with PixelFormat.PIXEL_FORMAT_RGB32 successfully into Emgu Image instances. When I try to use PixelFormat.PIXEL_FORMAT_DEPTH it generates a 'Parameter is not valid' exception in Emgu.

On the other hand, If I process the image in RGB32 I have no problem transforming into Emgu but, when I transform it back to PXCMImage, the depth values generated are off.

Any ideas would be appreciated. Thanks!

[Pol_P_]

To expand on the question, I printed some of the values obtained with the RGB32. This values come from the following chain of transformations:

PXCMImage -> Bitmap -> Emgu Image<Bgr, byte> -> Bitmap -> PXCMImage

To clarify, I am not doing any openCV operations on the converted image; just converting it back and forth.

Here are the values that I get for the point (200, 200) before (*1*) and after (*2*) transformation in 5 consecutive frames:

------------------------------------
*1* (200,200) > x: -357.7245, y: 95.45338, z: 1673
*2* (200,200) > x: -6484.37, y: 1730.256, z: 30326
------------------------------------
*1* (200,200) > x: -362.4285, y: 96.7086, z: 1695
*2* (200,200) > x: -6814.084, y: 1818.236, z: 31868
------------------------------------
*1* (200,200) > x: -367.5603, y: 98.07792, z: 1719
*2* (200,200) > x: -6484.37, y: 1730.256, z: 30326
------------------------------------
*1* (200,200) > x: -367.1326, y: 97.96381, z: 1717
*2* (200,200) > x: -6099.704, y: 1627.614, z: 28527
------------------------------------
*1* (200,200) > x: -352.3789, y: 94.02699, z: 1648
*2* (200,200) > x: -6539.323, y: 1744.92, z: 30583
------------------------------------
*1* (200,200) > x: -361.3594, y: 96.42332, z: 1690
*2* (200,200) > x: -6759.132, y: 1803.572, z: 31611
------------------------------------

Again, any thoughts are welcome! Thanks!