Use openCV with Intel Realsense SDK

Deepak_k_ · ‎01-06-2015

hi,

I want to do some image processing using depth and RGB stram of realsense camera. but i am not able to integrate opencv with realsense sdk. how can i receive depth frame in Mat variable of opencv so that i can do some processing in it.

thanks !!

Andradige_S_Intel · ‎01-06-2015

RealSense SDK doesn't provide frames in mat format, but you can easily convert to mat

first you access frame data as follows

// image is a PXCImage instance

PXCImage::ImageData data;

image->AcquireAccess(PXCImage::ACCESS_READ,&data);

Note: image planes are in data.planes[0-3] with pitch data.pitches[0-3]

Then you create an empty Mat using a constructor like this

Mat mat(int rows, int cols, int type)

after that you can copy data in to your mat from data above.

finally release access to data

image->ReleaseAccess(&data);

Deepak_k_ · ‎01-07-2015

how can i take video stream such as rgb, depth, ir in mat variable for processing.

thanks !!

Andradige_S_Intel · ‎01-07-2015

Well the type of stream you get by the AquireAccess() is based on the type of PXCImage you called that method on :)

So if you want get RGB image frame, for example then you do the following

psm = PXCSenseManager::CreateInstance();

then for each frame you do

PXCCapture::Sample *sample = psm->QuerySample();

PXCImage *rgbImage = sample->color;

PXCImage::ImageData frmData;

rgbImage->AcquireAccess(PXCImage::ACCESS_READ, PXCIMAGE::PIXEL_FORMAT_NV12, &frmData);

then you create a mat and copy the data over from frmData. Once you no longer need frmData, you just release the access

rgbImage->ReleaseAccess(&frmData);

Hope this helps

samontab · ‎01-07-2015

This is what I am currently using:

cv::Mat frameColor = cv::Mat::zeros(resolutionColor.height, resolutionColor.width, CV_8UC3);
cv::Mat frameDepth = cv::Mat::zeros(resolutionDepth.height, resolutionDepth.width, CV_32FC1);
cv::Mat frameIR = cv::Mat::zeros(resolutionIR.height, resolutionIR.width, CV_8UC1);

color stream:

PXCImage::PIXEL_FORMAT_RGB32, and then read the image as BGRA unsigned ints.

depth stream:
PXCImage::PIXEL_FORMAT_DEPTH_F32, and then read the image as floats.

IR stream:
PXCImage::PIXEL_FORMAT_Y8, and then read the image as unsigned chars.

Deepak_k_ · ‎01-08-2015

this is what i am doing to get depth stream.

/* Creates an instance of the PXCSenseManager */
PXCSenseManager *pp = PXCSenseManager::CreateInstance();
if (!pp)
{
wprintf_s(L"Unable to create the SenseManager\n");
return 3;
}

/* Collects command line arguments */
UtilCmdLine cmdl(pp->QuerySession());
if (!cmdl.Parse(L"-listio-nframes-sdname-csize-dsize-isize-file-record-noRender-mirror",argc,argv)) return 3;

   // Create stream renders
UtilRender renderd(L"Depth");
pxcStatus sts;
do {
       /* Apply command line arguments */
if (cmdl.m_dsize.size()>0)
       {
pp->EnableStream(PXCCapture::STREAM_TYPE_DEPTH, cmdl.m_dsize.front().first.width, cmdl.m_dsize.front().first.height, (pxcF32)cmdl.m_dsize.front().second);
}

       /* Sets file recording or playback */
       PXCCaptureManager *cm=pp->QueryCaptureManager();
       cm->SetFileName(cmdl.m_recordedFile, cmdl.m_bRecord);
       if (cmdl.m_sdname) cm->FilterByDeviceInfo(cmdl.m_sdname,0,0);

       if (cmdl.m_csize.size()==0 && cmdl.m_dsize.size()==0 && cmdl.m_isize.size()==0)
       {
PXCVideoModule::DataDesc desc={};
if (cm->QueryCapture())
           {
cm->QueryCapture()->QueryDeviceInfo(0, &desc.deviceInfo);
} else
           {
desc.deviceInfo.streams = PXCCapture::STREAM_TYPE_DEPTH;
}
pp->EnableStreams(&desc);

       }

/* Initializes the pipeline */
sts = pp->Init();

if (sts<PXC_STATUS_NO_ERROR) {
wprintf_s(L"Failed to locate any video stream(s)\n");
break;
}

/* Stream Data */
for (int nframes=0;nframes<cmdl.m_nframes;nframes++) {
/* Waits until new frame is available and locks it for application processing */
sts=pp->AcquireFrame(false);

if (sts<PXC_STATUS_NO_ERROR) {
if (sts==PXC_STATUS_STREAM_CONFIG_CHANGED) {
cerr << "Stream configuration was changed, re-initilizing\n" << endl;
pp->Close();
}
break;
}

           /* Render streams, unless -noRender is selected */
if (cmdl.m_bNoRender == false)
           {
const PXCCapture::Sample *sample = pp->QuerySample();
if (sample)
               {
if (sample->depth && !renderd.RenderFrame(sample->depth)) break;
               }
           }

/* Releases lock so pipeline can process next frame */
pp->ReleaseFrame();

if( _kbhit() ) { // Break loop
int c = _getch() & 255;
if( c == 27 || c == 'q' || c == 'Q') break; // ESC|q|Q for Exit
}
       }
   } while (sts == PXC_STATUS_STREAM_CONFIG_CHANGED);
           cerr <<"Exiting..."<<endl;

now where should i add

cv::Mat frameDepth = cv::Mat::zeros(resolutionDepth.height, resolutionDepth.width, CV_32FC1);

in the code so that i can get depth stream in Mat frameDepth.

i am not able to get depth stream in Mat variable.

can you please provide me full source code where mat access any stream . I am stucked from last 10 days into same place.

please help me

thanks !!

Rémy_B_ · ‎09-16-2015

You will find a solution here: http://stackoverflow.com/questions/32609341/comvert-a-pxcimage-into-an-opencv-mat/32609342

Regards,

Rémy

Lesly_Z_ · ‎08-11-2016

hey!

how do u get a good depth result with librealsense?

I am currently using this:

const uint16_t * depth_frame = reinterpret_cast<const uint16_t *>(dev->get_frame_data(rs::stream::depth));
cv::Mat depth16(360, 480, CV_16U, (void*)depth_frame);
cv::Mat depthM(depth16.size().height, depth16.size().width, CV_8UC1);
depth16.convertTo(depthM, CV_8UC1);

// min/max distance from the camera
unsigned short min = 0.5, max = 2.5;
cv::Mat img0 = cv::Mat::zeros(depthM.size().height, depthM.size().width, CV_8UC1);
cv::Mat depth_show;
double scale_ = 255.0 / (max-min);

depthM.convertTo(img0, CV_8UC1, scale_);
cv::applyColorMap(img0, depth_show, cv::COLORMAP_COOL);

cv::imshow("DEPTH", depth_show);

it returns a "depth image" from the video, but it is really noise with grains, not possible to see details of depth.

thanks