You may use pxcStatus

Christopher_C_ · ‎01-14-2015

I was wondering how I would be able to copy the webcam image stream from PXCsensemanager, allow the user to draw (on that frame), and then display the modified frame to the user since none of the OpenCV functions can take in a PXCsensemanager pointer as a parameter. I get how I'll have to delay the frame stream by 1 frame or more since I'll have to retrieve the frame, let the user modify it, then display the modified color stream afterwards. Also, would I need to create an transparent canvas on top of my color stream window? If so how would I do so?

Andradige_S_Intel · ‎01-14-2015

You may use pxcStatus AcquireAccess(Access access, PixelFormat format, Option options, ImageData *data); method to gain access to the frame's pixel level.

PXCImage::ImageData data;

// Access to the image data in the RGB32 format.

image->AcquireAccess(PXCImage::ACCESS_READ,PXCImage::PIXEL_FORMAT_RGB32,&data);

...

// work on the image plane data.planes[0] with pitch data.pitches[0].

...

// Release access after use.

image->ReleaseAccess(&data);

You may use memcpy() etc to initiate your buffers with frame pixels and eventually create openCV mats.

samontab · ‎01-15-2015

You should do this for every frame:

1. Read the frame from the camera as a PXCImage.
2. Convert PXCImage into cv::Mat
3. Process cv::Mat
4. Optionally convert cv::Mat into whatever other format you need for displaying, maybe back to PXCImage
5. Display image.
6. ???
7. Profit!!!

And you will not be 1 frame delayed as long as you can process fast enough each frame.
About your transparent canvas, they are your pixels, you can do whatever you want with them.

Jonathan_M_Intel · ‎01-15-2015

To reinforce the answers above, the best way to do this is:

1. Create an empty cv::Mat with the same dimensions, bit depth, and number of channels as the stream you're going to be copying from. (ex. CV_8UC3 for color (or CV_8UC4 for color with alpha) or CV_16UC1 for depth)

2. For every frame, once you acquire the PXCImage for the stream you want, you need to create a pixel data structure and pass that to the access function (starting to get complicated, but stay with me here).

3. Copy out your pixels from the data.planes[0] location to your desired Mat.data location.

4. Make sure you clean up and/or release everything.

Some version of the below code should work for you.

cv::Mat DepthMat(depthHeight,depthWidth,CV_16UC1);
cv::Mat InfraredMat(depthHeight,depthWidth,CV_8UC1);
cv::Mat ColorMat(colorHeight,colorWidth,CV_8UC3);



pxcStatus sts = SenseManager->AcquireFrame();
	if(PXC_STATUS_NO_ERROR == sts) {
		PXCCapture::Sample * sample = SenseManager->QuerySample();
		if(sample) {
			PXCImage * depthMap = sample->depth;
			PXCImage * irImage = sample->ir;
			PXCImage * colorImage = sample->color;
			PXCImage::ImageData data;


			if(depthMap) {
				sts = depthMap->AcquireAccess(PXCImage::ACCESS_READ, &data);
				if(PXC_STATUS_NO_ERROR == sts) {
					//Copy from data.planes[0] to DepthMat.data					
 					memcpy(DepthMat.data,data.planes[0],sizeof(unsigned short) * depthWidth * depthHeight);

					depthMap->ReleaseAccess(&data);
				}



			}
			if(irImage) {
				sts = irImage->AcquireAccess(PXCImage::ACCESS_READ,&data);
				if(PXC_STATUS_NO_ERROR == sts) {
 					//Copy from data.planes[0] to InfraredMat.data	
 					memcpy(InfraredMat.data,data.planes[0],sizeof(unsigned char) * depthWidth * depthHeight);
					irImage->ReleaseAccess(&data);
				}
			}

		
			if(colorImage) {
				sts = colorImage->AcquireAccess(PXCImage::ACCESS_READ,PXCImage::PIXEL_FORMAT_RGB24,&data);
				if(PXC_STATUS_NO_ERROR == sts) {
 					//Copy from data.planes[0] to ColorMat.data					
 					memcpy(ColorMat.data,data.planes[0],sizeof(unsigned char) * 3 * colorWidth * colorHeight);

					colorImage->ReleaseAccess(&data);
				}
			}
		}
		SenseManager->ReleaseFrame();

	}

Christopher_C_ · ‎01-19-2015

Jonathan T. (Intel) wrote:

To reinforce the answers above, the best way to do this is:

1. Create an empty cv::Mat with the same dimensions, bit depth, and number of channels as the stream you're going to be copying from. (ex. CV_8UC3 for color (or CV_8UC4 for color with alpha) or CV_16UC1 for depth)

2. For every frame, once you acquire the PXCImage for the stream you want, you need to create a pixel data structure and pass that to the access function (starting to get complicated, but stay with me here).

3. Copy out your pixels from the data.planes[0] location to your desired Mat.data location.

4. Make sure you clean up and/or release everything.

Some version of the below code should work for you.
 
cv::Mat DepthMat(depthHeight,depthWidth,CV_16UC1);
cv::Mat InfraredMat(depthHeight,depthWidth,CV_8UC1);
cv::Mat ColorMat(colorHeight,colorWidth,CV_8UC3);



pxcStatus sts = SenseManager->AcquireFrame();
	if(PXC_STATUS_NO_ERROR == sts) {
		PXCCapture::Sample * sample = SenseManager->QuerySample();
		if(sample) {
			PXCImage * depthMap = sample->depth;
			PXCImage * irImage = sample->ir;
			PXCImage * colorImage = sample->color;
			PXCImage::ImageData data;


			if(depthMap) {
				sts = depthMap->AcquireAccess(PXCImage::ACCESS_READ, &data);
				if(PXC_STATUS_NO_ERROR == sts) {
					//Copy from data.planes[0] to DepthMat.data					
 					memcpy(DepthMat.data,data.planes[0],sizeof(unsigned short) * depthWidth * depthHeight);

					depthMap->ReleaseAccess(&data);
				}



			}
			if(irImage) {
				sts = irImage->AcquireAccess(PXCImage::ACCESS_READ,&data);
				if(PXC_STATUS_NO_ERROR == sts) {
 					//Copy from data.planes[0] to InfraredMat.data	
 					memcpy(InfraredMat.data,data.planes[0],sizeof(unsigned char) * depthWidth * depthHeight);
					irImage->ReleaseAccess(&data);
				}
			}

		
			if(colorImage) {
				sts = colorImage->AcquireAccess(PXCImage::ACCESS_READ,PXCImage::PIXEL_FORMAT_RGB24,&data);
				if(PXC_STATUS_NO_ERROR == sts) {
 					//Copy from data.planes[0] to ColorMat.data					
 					memcpy(ColorMat.data,data.planes[0],sizeof(unsigned char) * 3 * colorWidth * colorHeight);

					colorImage->ReleaseAccess(&data);
				}
			}
		}
		SenseManager->ReleaseFrame();

	}

I can see what you're doing here, but don't I want a transparent canvas over the webcam stream? This is because if I tried to implement an erase function, then how would I be able to get rid of user made markings on the stream without erasing parts of the stream itself?

Also, from your code, would I try to manipulate the image during this part:

			if(colorImage) {
				sts = colorImage->AcquireAccess(PXCImage::ACCESS_READ,PXCImage::PIXEL_FORMAT_RGB24,&data);
				if(PXC_STATUS_NO_ERROR == sts) {
 					//Copy from data.planes[0] to ColorMat.data					
 					memcpy(ColorMat.data,data.planes[0],sizeof(unsigned char) * 3 * colorWidth * colorHeight);

					colorImage->ReleaseAccess(&data);
				}

I'm assuming I manipulate the freshly copied data.planes (which is now stored in ColorMat.data) and display it. However, would I need to convert that image back to a PXCImage type before rendering it? If not, then do I just render the ColorMat frame after manipulating it?

Christopher_C_ · ‎01-19-2015

samontab wrote:

3. Process cv::Mat
4. Optionally convert cv::Mat into whatever other format you need for displaying, maybe back to PXCImage

And you will not be 1 frame delayed as long as you can process fast enough each frame.
About your transparent canvas, they are your pixels, you can do whatever you want with them.

Thanks for the help! The problem is that I'm not sure how would I convert the image back from the colorMat.data to the PXCImage::*colorIm after manipulating it (which function would I use?). Another approach I was thinking is that I would just manipulate the colorMat.data then just render that frame without converting it back to a PXCImage type (it seem like that would be a better option since I would not have to waste time converting). However, when I try the latter, I get an error. This is what I try (this builds off of Jonathan's code):

if (PXC_STATUS_NO_ERROR == sts) {
			if (sample) {
				PXCImage * colorImage = colorIm;
				PXCImage::ImageData data;
				if (colorImage) {
					sts = colorImage->AcquireAccess(PXCImage::ACCESS_READ, PXCImage::PIXEL_FORMAT_RGB24, &data);
					if (PXC_STATUS_NO_ERROR == sts) {
						//Copy from data.planes[0] to ColorMat.data					
						memcpy(colorMat.data, data.planes[0], sizeof(unsigned char)* 3 * colorWidth * colorHeight);

                                                cvshow("Webcam Stream", colorMat.data);
						colorImage->ReleaseAccess(&data);
					}
				}
			}

In this example, I test out the cvshow function without manipulating the ColorMat.data, but when I attempt to do this, I get an error

Jonathan_M_Intel · ‎01-19-2015

Can you share more of your code? It may be that you have either not allocated enough memory in the destination Mat or you have the wrong (too large) values for your colorWidth and colorHeight, which in either case would cause you to be attempting to read or write unallocated memory.

Christopher_C_ · ‎01-19-2015

Jonathan T. (Intel) wrote:

Can you share more of your code? It may be that you have either not allocated enough memory in the destination Mat or you have the wrong (too large) values for your colorWidth and colorHeight, which in either case would cause you to be attempting to read or write unallocated memory.

I sent you a private message with my code and some questions. Thanks for the assistance!

Overlaying images with OpenCV on color webcam stream