topic Re: Difference between PXCImage view and cv::Mat view for the depth stream in Items with no label

Difference between PXCImage view and cv::Mat view for the depth stream

lluci1 — Wed, 26 Apr 2017 09:58:37 GMT

Hello

I am working with the RealSense R200 camera using Visual Studio & C++.

I need to use OpenCV to do image processing so I converted the PXCImage * into cv::Mat.

But, as you can see, the two images are different (like if I am loosing information).

Does anybody can help me?

Render display screenshot Imshow display screenshot

Thanks,

Lucie

Re: Difference between PXCImage view and cv::Mat view for the depth stream

idata — Wed, 26 Apr 2017 20:14:35 GMT

Hi Lucie,

Thank you for contacting us.

We've investigated about your case and found two useful links that relate to this case. The first one is a community thread where the user has a similar question and the second one is a guide to use LibRealSense and OpenCV to stream RGB and Depth Data. The latter has a complete example that explains how to use LibRealSense and OpenCV to display both an RGB stream as well as Depth stream into two separate OpenCV created windows.

https://software.intel.com/en-us/forums/realsense/topic/599415

https://software.intel.com/en-us/articles/using-librealsense-and-opencv-to-stream-rgb-and-depth-data

Let us know if you find this useful.

Regards,

-Sergio A

Re: Difference between PXCImage view and cv::Mat view for the depth stream

lluci1 — Thu, 27 Apr 2017 09:29:54 GMT

Hi Sergio A,

Thank you for you answer ,

Unfortunately, I already looked those two links and I am still blocked. It might be a format problem or I don't know..

This is my code (a mix of what I found on internet), if someone can help me :

# include "opencv2/core.hpp"

# include "opencv2/highgui.hpp"

# include

# include "pxcsensemanager.h"

# include "pxcmetadata.h"

# include "util_cmdline.h"

# include "util_render.h"

# include

using namespace std;

using namespace cv;

void RecordORPlayback(pxcCHAR *file, bool record) {

// Create a SenseManager instance

PXCSenseManager *sm = PXCSenseManager::CreateInstance();

PXCSession *session;

// Set file recording or playback

sm->QueryCaptureManager()->SetFileName(file, record);

// Select the color stream

sm->EnableStream(PXCCapture::STREAM_TYPE_DEPTH, 320, 240, 30);

// Initialize and Record 300 frames

sm->Init();

for (int i = 0; i<100; i++) {

// This function blocks until a color sample is ready

if (sm->AcquireFrame(true)

// Retrieve the sample

PXCCapture::Sample *sample = sm->QuerySample();

// Go fetching the next sample

sm->ReleaseFrame();

}

// close down

sm->Release();

}

cv::Mat PXCImage2CVMat(PXCImage *pxcImage, PXCImage::PixelFormat format)

{

PXCImage::ImageData data;

pxcImage->AcquireAccess(PXCImage::ACCESS_READ, format, &data);

int width = pxcImage->QueryInfo().width;

int height = pxcImage->QueryInfo().height;

if (!format)

format = pxcImage->QueryInfo().format;

int type;

if (format == PXCImage::PIXEL_FORMAT_Y8)

type = CV_8UC1;

else if (format == PXCImage::PIXEL_FORMAT_RGB24)

type = CV_8UC3;

else if (format == PXCImage::PIXEL_FORMAT_DEPTH_F32)

type = CV_32F;

cv::Mat ocvImage = cv::Mat(cv::Size(width, height), type, data.planes[0], data.pitches[0]);

pxcImage->ReleaseAccess(&data);

return ocvImage;

}

int main(int argc, WCHAR* argv[]) {

//Define some parameters for the camera

cv::Size frameSize = cv::Size(320, 240);

float frameRate = 30;

cv::Mat frameColor = cv::Mat::zeros(frameSize, CV_8UC3);

cv::Mat frameDepth = cv::Mat::zeros(frameSize, CV_32F);

/* Creates an instance of the PXCSenseManager */

PXCSenseManager *pp = PXCSenseManager::CreateInstance();

if (!pp) {

wprintf_s(L"Unable to create the SenseManager\n");

return 3;

}

/* Sets file recording or playback */

PXCCaptureManager *cm = pp->QueryCaptureManager();

pxcCHAR *filename = (L"DEPTHRenderTEST");

RecordORPlayback(filename, true);

// Create stream renders

UtilRender renderc(L"Color"), renderd(L"Depth"), renderi(L"IR"), renderr(L"Right"), renderl(L"Left");

pxcStatus sts;

do {

/* Apply command line arguments */

pxcBool revert = false;

PXCVideoModule::DataDesc desc = {};

desc.deviceInfo.streams = PXCCapture::STREAM_TYPE_COLOR | PXCCapture::STREAM_TYPE_DEPTH;

revert = true;

pp->EnableStreams(&desc);

/* Initializes the pipeline */

sts = pp->Init();

if (sts

if (revert) {

/* Enable a single stream */

pp->Close();

pp->EnableStream(PXCCapture::STREAM_TYPE_DEPTH);

sts = pp->Init();

if (sts

pp->Close();

pp->EnableStream(PXCCapture::STREAM_TYPE_COLOR);

sts = pp->Init();

}

if (sts

wprintf_s(L"Failed to locate any video stream(s)\n");

pp->Release();

return sts;

}

/* Reset all properties */

PXCCapture::Device *device = pp->QueryCaptureManager()->QueryDevice();

device->ResetProperties(PXCCapture::STREAM_TYPE_ANY);

for (;;) {

// for (int i=0;i<51;i++){

/* Waits until new frame is available and locks it for application processing */

sts = pp->AcquireFrame(true);

if (sts

if (sts == PXC_STATUS_STREAM_CONFIG_CHANGED) {

wprintf_s(L"Stream configuration was changed, re-initilizing\n");

pp->Close();

}

break;

}

const PXCCapture::Sample *sample = pp->QuerySample();

if (sample) {

if (sample->depth && !renderd.RenderFrame(sample->depth)) break;

if (sample->depth) {

frameDepth = PXCImage2CVMat(sample->depth, PXCImage::PIXEL_FORMAT_DEPTH_F32);

/* string result_depthCV = "DepthOpenCV.png";

imwrite(result_depthCV, frameDepth);*/

namedWindow("Depth", CV_WINDOW_NORMAL);

imshow("Depth", frameDepth);

waitKey(1);

}

if (sample->color) {

frameColor = PXCImage2CVMat(sample->color, PXCImage::PIXEL_FORMAT_RGB24);

namedWindow("Color", CV_WINDOW_NORMAL); //works without this line too

imshow("Color", frameColor);

waitKey(1);

}

/* Releases lock so pipeline can process next frame */

pp->ReleaseFrame();

}

if (_kbhit()) { // Break loop

int c = _getch() & 255;

if (c == 27 || c == 'q' || c == 'Q') break; // ESC|q|Q for Exit

}

} while (sts == PXC_STATUS_STREAM_CONFIG_CHANGED);

wprintf_s(L"Exiting\n");

// Clean Up

pp->Release();

system("Pause");

return 0;

}

Thanks,

Lucie

Re: Difference between PXCImage view and cv::Mat view for the depth stream

idata — Thu, 27 Apr 2017 20:35:47 GMT

Hi Lucie,

We looked online and even though we didn't find any official guides that might help you with the code, we did found a couple of links that may be helpful

https://software.intel.com/en-us/forums/realsense/topic/624897

http://stackoverflow.com/questions/37266390/how-to-convert-realsense-rgb-frame-to-cvmat-in-ubuntu-or-other-linux-env

From the looks of both pictures, they're obviously different, but, the depth patterns don't seem to be so different from each other.

It looks as if one of the images needs further calibration in order to look the same as the first image.

I looked at some examples and found that frameColor and frameDepth used this way solved the issue for some customers:

cv::Mat frameColor = cv::Mat::zeros(resolutionColor.height, resolutionColor.width, CV_8UC3);

cv::Mat frameDepth = cv::Mat::zeros(resolutionDepth.height, resolutionDepth.width, CV_32FC1);

Keep us updated.

-Sergio A

Re: Difference between PXCImage view and cv::Mat view for the depth stream

idata — Tue, 02 May 2017 19:49:14 GMT

Hi Lucie,

Do you still need assistance with this case? We'll be waiting for your response.

-Sergio A

Re: Difference between PXCImage view and cv::Mat view for the depth stream

lluci1 — Wed, 03 May 2017 10:18:35 GMT

Hi Sergio,

Thanks for your interest in my issue.

I just solved it

Finally, I used :

cv::Mat frameDepth = cv::Mat::zeros(frameSize, CV_16U);

cv::Mat frameDepth8b = cv::Mat::zeros(frameSize, CV_8UC1);

and then,

if (sample->depth) {

frameDepth = PXCImage2CVMat(sample->depth, PXCImage::PIXEL_FORMAT_DEPTH);

frameDepth.convertTo(frameDepth8b, CV_8UC1, 255.0 / 4096.0);

// imwrite

string result_depthCV = "DepthOpenCV.png";

imwrite(result_depthCV, frameDepth8b, { CV_IMWRITE_PNG_STRATEGY_DEFAULT });

// imshow

namedWindow("Depth", CV_WINDOW_AUTOSIZE);

imshow("Depth", frameDepth8b);

waitKey(1);

}

I don't really understand with I have to convert frameDepth in CV_8UC1 because normally imwrite can save an 16-bit image in PNG format. But it now at least, it works

Thanks again ,

Lucie

Re: Difference between PXCImage view and cv::Mat view for the depth stream

idata — Wed, 03 May 2017 19:48:18 GMT

Hi Lucie,

Thank you for confirming you've fixed the issue and sharing your results with the community. Don't hesitate to come back if help is needed, we'd be happy to assist you.

Regards,

-Sergio A

Re: Difference between PXCImage view and cv::Mat view for the depth stream

秦文翔 — Wed, 02 Aug 2017 13:06:32 GMT

Hi Lucie

You said you have solved this problem, but when I used your method, it didn't work. As you can see:

It is still different.

I hope I will receive your reply.