how to get the aligned depth and color data

[Binbin_X_]

I set both the depth and color streams to 640*480, but they are not aligned, for example, the depth of (320,240) in color stream is not the value of (320, 240) in depth stream.
I tried pxcprojection, but I can not get correct value, can someone help me? I have tried projection->QueryVertiecs(image_depth, vertices), but the values in vertices are all 0, and tried MapColorToDepth(image_depth, 640 * 480, ij, pos_uv); but the value in pos_uv are all -1,

can someone provide me a sample code to fix my problem? 

int width = 640;
int height = 480;
PXCSenseManager *sm = PXCSenseManager::CreateInstance();

sm->EnableStream(PXCCapture::STREAM_TYPE_COLOR, width, height);
sm->EnableStream(PXCCapture::STREAM_TYPE_DEPTH, width, height);

sm->Init();

UtilRender color_render(L"Color Stream");
UtilRender depth_render(L"Depth Stream");
PXCCaptureManager *g_captureManager = sm->QueryCaptureManager();
PXCCapture::Device *g_device = g_captureManager->QueryDevice();
PXCProjection *projection = g_device->CreateProjection();
PXCPoint3DF32 *vertices = new PXCPoint3DF32[640 * 480];

while (!quit)
{
    pxcStatus sts = sm->AcquireFrame(true);
    if (sts < PXC_STATUS_NO_ERROR) {
        std::cout << "Unable to acquire the frames!" << std::endl;
        break;
    }
    if (sm->AcquireFrame(true)<PXC_STATUS_NO_ERROR) break;
    PXCCapture::Sample *sample = sm->QuerySample();
    
    PXCImage *image_color = sample->color;
    PXCImage *image_depth = sample->depth;
    
    PXCImage::ImageData data_color;
    PXCImage::ImageData data_depth;
    
    projection->QueryVertices(image_depth, vertices);
    //projection->MapColorToDepth(image_depth, 640 * 480, ij, pos_uv);
    image_color->AcquireAccess(PXCImage::ACCESS_READ, PXCImage::PIXEL_FORMAT_RGB32, &data_color);
    image_depth->AcquireAccess(PXCImage::ACCESS_READ_WRITE, PXCImage::PIXEL_FORMAT_DEPTH, &data_depth);
    
    // do something

    image_color->ReleaseAccess(&data_color);
    image_depth->ReleaseAccess(&data_depth);
    sm->ReleaseFrame();
}

sm->Close();

[samontab]

The two cameras cover different areas in the real world. You need to project one into the other to have a matching image. You were right with pxcprojection.

Take a look at the samples, there is one that shows you how to work with projections.

[Binbin_X_]

samontab wrote:

The two cameras cover different areas in the real world. You need to project one into the other to have a matching image. You were right with pxcprojection.

Take a look at the samples, there is one that shows you how to work with projections.

can you tell me which sample? I can not find one that work with projection

[samontab]

There is a newer, more complete example in the latest SDK, R2. Just run the samples browser and it will be there... can't tell you exactly the name of it since I am in Linux at the moment...

[Marios_B_]

Binbin X. wrote:

I set both the depth and color streams to 640*480, but they are not aligned, for example, the depth of (320,240) in color stream is not the value of (320, 240) in depth stream.
I tried pxcprojection, but I can not get correct value, can someone help me? I have tried projection->QueryVertiecs(image_depth, vertices), but the values in vertices are all 0, and tried MapColorToDepth(image_depth, 640 * 480, ij, pos_uv); but the value in pos_uv are all -1,

can someone provide me a sample code to fix my problem? 

int width = 640;
int height = 480;
PXCSenseManager *sm = PXCSenseManager::CreateInstance();

sm->EnableStream(PXCCapture::STREAM_TYPE_COLOR, width, height);
sm->EnableStream(PXCCapture::STREAM_TYPE_DEPTH, width, height);

sm->Init();

UtilRender color_render(L"Color Stream");
UtilRender depth_render(L"Depth Stream");
PXCCaptureManager *g_captureManager = sm->QueryCaptureManager();
PXCCapture::Device *g_device = g_captureManager->QueryDevice();
PXCProjection *projection = g_device->CreateProjection();
PXCPoint3DF32 *vertices = new PXCPoint3DF32[640 * 480];

while (!quit)
{
    pxcStatus sts = sm->AcquireFrame(true);
    if (sts < PXC_STATUS_NO_ERROR) {
        std::cout << "Unable to acquire the frames!" << std::endl;
        break;
    }
    if (sm->AcquireFrame(true)<PXC_STATUS_NO_ERROR) break;
    PXCCapture::Sample *sample = sm->QuerySample();
    
    PXCImage *image_color = sample->color;
    PXCImage *image_depth = sample->depth;
    
    PXCImage::ImageData data_color;
    PXCImage::ImageData data_depth;
    
    projection->QueryVertices(image_depth, vertices);
    //projection->MapColorToDepth(image_depth, 640 * 480, ij, pos_uv);
    image_color->AcquireAccess(PXCImage::ACCESS_READ, PXCImage::PIXEL_FORMAT_RGB32, &data_color);
    image_depth->AcquireAccess(PXCImage::ACCESS_READ_WRITE, PXCImage::PIXEL_FORMAT_DEPTH, &data_depth);
    
    // do something

    image_color->ReleaseAccess(&data_color);
    image_depth->ReleaseAccess(&data_depth);
    sm->ReleaseFrame();
}

sm->Close();

Hey Binbin,did you find any solution? I also want to do the same thing.If yes, please post a code sample here please!

[Lesly_Z_]

How can I achieve this with librealsense and something similar to: buffers[1].show(dev, rs::stream::color_aligned_to_depth, s, 0, s, h-h/2);

thanks!

[samontab]

The documentation of librealsense is pretty good. Have a look there:

https://github.com/IntelRealSense/librealsense/blob/master/doc/projection.md