Accessing R200 Infrared Calibration Data (in Unity)

[SMali10]

Hello!

I've recently received my R200 dev kit and so far it's pretty great. However I am having trouble with the "QueryStreamProjectionParameters" inside a Unity C# script. Here is my initialization code:

PXCMSenseManager psm = PXCMSenseManager.CreateInstance();

if (psm == null) {
	Debug.LogError ("SenseManager Initialization Failed");
	return;
}

psm.EnableStream(PXCMCapture.StreamType.STREAM_TYPE_LEFT, 628, 468, 60f);
psm.EnableStream(PXCMCapture.StreamType.STREAM_TYPE_RIGHT, 628, 468, 60f);

pxcmStatus sts = psm.Init();
if (sts != pxcmStatus.PXCM_STATUS_NO_ERROR) {
	Debug.LogError ("PXCMSenseManager.Init Failed");
	OnDisable (); // Clean-up
	return;
}

PXCMCapture.Device camera = psm.captureManager.device;

PXCMProjection projection = camera.CreateProjection();
PXCMCalibration calib = projection.QueryInstance<PXCMCalibration>();

PXCMCalibration.StreamCalibration leftCalib = new PXCMCalibration.StreamCalibration();
PXCMCalibration.StreamTransform leftTrans = new PXCMCalibration.StreamTransform();

calib.QueryStreamProjectionParameters (PXCMCapture.StreamType.STREAM_TYPE_LEFT, out leftCalib, out leftTrans);

projection.Dispose();

I am trying to access the calibration data for the left and right infrared cameras. Everything works great up until "QueryStreamProjectionParameters". Everytime I try that function the Unity editor crashes, no matter what stream I try and select. Otherwise the camera works fine and I am able to access the raw infrared frames.

Please let me know if it is possible to access the calibration data for these cameras, or if I am doing something wrong. Thank you!

[MartyG]

Whilst I don't know anything about accessing calibration data, I did notice that your C# script seemed to lack the block of "header" code at the top of it that Unity C# scripts usually need.  Maybe it would make a difference if you included that in your script, if you have not done so already.

The default header code is usually:

using UnityEngine;
using System.Collections;

All of the Action camera scripts packaged with the Unity SDK Toolkit add a couple of extra lines to the header:

using UnityEngine;
using System.Collections;
using System.Collections.Generic;
using RSUnityToolkit; 

Edit: I had a go at trying to get the script working using a test object in my project.  Before I post my adapted script, I'll point out a couple of things I noted:

* The script seems to be calling an OnDisable() function to perform clean-up when the Sense Manager does not initialize, but the script does not have an OnDisable function defined and so the script cannot find it.  For example, OnDisable is normally used in the following format to define a block of code:  

void onDisable();

This makes me wonder if the script you posted was part of a set of more than one script, one of which contains an OnDisable() function?  I deleted this line from my test script, as my goal was to get it running at all rather than have it working perfectly.

*  At the end of your psm.EnableStream statements you used the number '60f'.  The f tells Unity to treat the number as a "float" type number and is often used with decimal numbers like 0.5, as a script won't run if it has a decimal number without an f on the end of it.  Whole numbers like 60 do not seem to need to be defined as a float number, so I removed the f in my version of the script.

*  Whilst you have defined statements for left-side calibration, there are no statements in the script for right-side calibration.  I don't know if this is intentional.

When I ran the script, it didn't crash out but I got the message every time that PXCMSenseManager could not be initialized.  So the program ran but without the SenseManager init, the code did not do anything.

using UnityEngine;
using System.Collections;
using System.Collections.Generic;
using RSUnityToolkit; 

public class tester : MonoBehaviour {
	
	void Start () {

		PXCMSenseManager psm = PXCMSenseManager.CreateInstance();
		
		if (psm == null) {
			Debug.LogError ("SenseManager Initialization Failed");
			return;
		}


		psm.EnableStream(PXCMCapture.StreamType.STREAM_TYPE_LEFT, 628, 468, 60);
		psm.EnableStream(PXCMCapture.StreamType.STREAM_TYPE_RIGHT, 628, 468, 60);
		
		pxcmStatus sts = psm.Init();  
		if (sts != pxcmStatus.PXCM_STATUS_NO_ERROR) {
			Debug.LogError ("PXCMSenseManager.Init Failed");
	
			return;
		}
		
		PXCMCapture.Device camera = psm.captureManager.device;
		
		PXCMProjection projection = camera.CreateProjection();
		PXCMCalibration calib = projection.QueryInstance<PXCMCalibration>();
		
		PXCMCalibration.StreamCalibration leftCalib = new PXCMCalibration.StreamCalibration();
		PXCMCalibration.StreamTransform leftTrans = new PXCMCalibration.StreamTransform();
		
		calib.QueryStreamProjectionParameters (PXCMCapture.StreamType.STREAM_TYPE_LEFT, out leftCalib, out leftTrans);
		
		projection.Dispose();

	
	}
}

Note: if you try using the above script, change the word 'tester' in the line 

public class tester : MonoBehaviour {

to the file-name of your own C# script.

[SMali10]

Hi Marty,

Thanks so much for going through the trouble and actually trying it in Unity, I really appreciate it!

Unfortunately I have to apologize. I should have been clearer when I said that this was just my initialization code. I could not post the full script so this was just a snippet of my start function.

Since there is not a lot of information on accessing the raw streams from the R200, here is a Unity script that access the raw 16bit data from the left IR camera.

using UnityEngine;
using System.Collections;
using UnityEngine.UI;
using RSUnityToolkit;

public class irStream : MonoBehaviour {
	private PXCMSenseManager psm;

	private Texture2D irTexture;
	private ushort[] irData;
	private byte[] imgData;

	public RawImage displayImage;

	void Start () {
		psm = PXCMSenseManager.CreateInstance();
		if (psm == null) {
			Debug.LogError ("SenseManager Initialization Failed");
			return;
		}

		psm.EnableStream(PXCMCapture.StreamType.STREAM_TYPE_LEFT, 628, 468, 60f);
		// FUN FACT: in order to access the right stream of the R200 you must enable the left stream first
		//psm.EnableStream(PXCMCapture.StreamType.STREAM_TYPE_RIGHT, 628, 468, 60f);

		pxcmStatus sts = psm.Init();
		if (sts != pxcmStatus.PXCM_STATUS_NO_ERROR) {
			Debug.LogError ("PXCMSenseManager.Init Failed");
			OnDisable (); // Clean-up
			return;
		}

		irData = new ushort[640*468];
		imgData = new byte[628*468*4];
	}

	void Update () {
		if (psm == null) return;
		if (psm.AcquireFrame(true) != pxcmStatus.PXCM_STATUS_NO_ERROR) return;

		PXCMCapture.Sample sample = psm.QuerySample();

		if (sample != null)
		{
			PXCMImage irImageSource = sample.left;

			if (irImageSource != null)
			{
				// Create the Texture 2D object if it doesn't exist already
				if (irTexture == null)
				{
					irTexture = new Texture2D(irImageSource.info.width,
					                              irImageSource.info.height,
					                               TextureFormat.RGBA32, false);
					displayImage.texture = irTexture;
				}

				// Get the raw 16bit data from the infrared image
				PXCMImage.ImageData irImageData;
				irImageSource.AcquireAccess(PXCMImage.Access.ACCESS_READ,
				                        PXCMImage.PixelFormat.PIXEL_FORMAT_Y16,
				                        out irImageData);
				irImageData.ToUShortArray(0, irData);

				// despite the ir camera being 628 pixels wide, the raw bytes are stored with a "pitch" of 640
				// in the case of 16bit data, the pitch is returned as 1280
				int pitch = irImageData.pitches[0]/2;
				int width = irImageSource.info.width;
				int height = irImageSource.info.height;
				
				for(int y = 0; y < height; y++){
					for(int x = 0; x < width; x++){
						int iA = (height-y-1) * pitch + x;
						int iB = (y * width + x) * 4;
						
						byte intensity = (byte)(irData[iA]/4);// raw pixels range between 0-1024

						imgData[iB] = intensity;
						imgData[iB+1] = intensity;
						imgData[iB+2] = intensity;
						imgData[iB+3] = 255; // Alpha
					}
				}
				
				irTexture.LoadRawTextureData(imgData);
				irTexture.Apply();
				
				irImageSource.ReleaseAccess(irImageData);
			}
		}
		psm.ReleaseFrame();
	}

	void OnDisable()
	{
		if (psm == null) return;
		psm.Dispose();
	}
}

Just attach the script to a game object, create a "RawImage" UI object (in Unity 4.6+) and attach that object to the "displayImage" property.

Back to the original question, it really is just the function "QueryStreamProjectionParameters " that causes a fatal error. There is no error message displayed or log to check. The application just shuts down immediately.

This code goes at the end of the start function. If the function in question is uncommented Unity crashes on my system.

PXCMCapture.Device camera = psm.captureManager.device;

PXCMProjection projection = camera.CreateProjection();
PXCMCalibration calib = projection.QueryInstance<PXCMCalibration>();

PXCMCalibration.StreamCalibration leftCalib = new PXCMCalibration.StreamCalibration();
PXCMCalibration.StreamTransform leftTrans = new PXCMCalibration.StreamTransform();

//calib.QueryStreamProjectionParameters (PXCMCapture.StreamType.STREAM_TYPE_LEFT, out leftCalib, out leftTrans);

projection.Dispose();

I have noticed a few other functions that can cause Unity to crash. For example "EnableStream" will crash if any unsupported variables are entered as the resolution or framerate. However even this returns an error message.

Other Stuff!

30, 60 and 90 are definitely fine to enter without the float declaration. In the documentation it says this value should be a float, and since Unity is picky when it comes to declaring doubles and floats I just do it out of habit, even for whole numbers.

I haven't added the right side calibration as it didn't make sense to call a fatal function after the program has already crashed :)

Again, thank you, I appreciate you taking the time.

[MartyG]

Yeah, I thought that the absence of the right-hand calibration code was probably due to you making sure the left-side code worked first.  I work the same way when I'm testing something new.

I'm a C# novice, having only started learning it a couple of weeks ago after having been a JavaScript meister up til then, so this advanced stuff is still a bit outside of my sphere of experience.  By comparing your code to the RealSense manual's entry for PXCMCalibration though, I was able to see what you're trying to do with your code (I found the initialization routine you used, with additional notations in the manual's version to help me understand it).

I went through numerous online examples of PXCMProjection.  I noticed that all of them used variations of the PXCMProjection Projection line.  The key difference between their code and yours is that they use:

PXCMProjection projection = device.CreateProjection();

Whereas you replace 'device' with 'camera'.

PXCMProjection projection = camera.CreateProjection();

Every time I google for a version of the line that includes 'camera', I get taken only to ones that use 'device'.  So maybe you need to change 'camera' to 'device' to get it to stop crashing.

[SMali10]

I did try:

PXCMProjection projection = PXCMCapture.Device.CreateProjection();

However there is no static method for CreateProjection. The function has to be used with an instance of the device class, and not the device class itself.

The line:

PXCMProjection projection = psm.captureManager.device.CreateProjection();

is equivalent to:

PXCMCapture.Device camera = psm.captureManager.device;

PXCMProjection projection = camera.CreateProjection();

"camera" is simply a reference to "psm.captureManager.device", so that it's easier to reference the device later on. It could be that I am missing something in the initialization of the projection or the calibration. I do know that other functions work by accessing the device through a reference variable such as exposure settings.

Thanks!

[MartyG]

I'm probably at the limits of my current level of knowledge of C# now.  I know there are helpful, friendly developers on this forum who are far more familiar with writing in the language than I am though, so hopefully they'll be able to help you find a solution quickly.  Best of luck!

[SMali10]

No worries, I appreciate the help!

[SMali10]

I tried to access just the color calibration of the R200 using the C# "DF_RawStreams" example. The good news is that the function "QueryStreamProjectionParameters" doesn't cause the application to crash. The bad news is that it doesn't seem to be supported?

I added this to "RawStreams.cs" just after initialzation. In order to to get "SetStatus" to display the optional "ShowTickPerformance" has to be disabled.

SetStatus("Init Started");
if (sm.Init() >= pxcmStatus.PXCM_STATUS_NO_ERROR)
{
/* Reset all properties */
sm.captureManager.device.ResetProperties(PXCMCapture.StreamType.STREAM_TYPE_ANY);

// Start Find Calibration Code ------ New Code
PXCMProjection projection = sm.captureManager.device.CreateProjection();
PXCMCalibration calib = projection.QueryInstance<PXCMCalibration>();

PXCMCalibration.StreamCalibration calibData;
PXCMCalibration.StreamTransform calibTrans;

SetStatus(calib.QueryStreamProjectionParameters(PXCMCapture.StreamType.STREAM_TYPE_COLOR, out calibData, out calibTrans).ToString());
                    
projection.Dispose();
// End Find Calibration Code ------ End New Code

/* Set mirror mode */
PXCMCapture.Device.MirrorMode mirror = Mirror ? PXCMCapture.Device.MirrorMode.MIRROR_MODE_HORIZONTAL : PXCMCapture.Device.MirrorMode.MIRROR_MODE_DISABLED;
sm.captureManager.device.SetMirrorMode(mirror);

//SetStatus("Streaming");

As far as I can tell this follows the documentation. However the function "QueryStreamProjectionParameters" returns "PXCM_STATUS_FEATURE_UNSUPPORTED".

Meanwhile "QueryStreamProjectionParametersEx" returns "PXCM_STATUS_DATA_UNAVAILABLE".

This is for every available stream. Am I missing something here? Are the functions broken? Yet to be implemented? Or only intended for the F200?

Any help would be appreciated. Thanks!

[SMali10]

Alternatively I've tried to gather the rectified IR calibration data, using the "QueryDepth..." functions of the "PXCMCapture.Device" interface.

I'm going to make a lot of assumptions here. Assuming that the IR cameras are rectified using a similar function to OpenCV's "StereoRectify" function, the focal lengths, and vertical principle points of each camera should be identical. These can be easily accessed with "QueryDepthFocalLength" and "QueryDepthPrincipalPoint".

The documentation states that these are "generic" model values and not the actual calibrated data. Not ideal, but it's better than nothing. All of these functions work great, as long as a depth or ir stream is enabled.

So that leaves the horizontal principle point of the right camera, and the horizontal disparity shift between the cameras.

My hope is that the principle points of both ir cameras are the same, and that the function "QueryDSDisparityShift" returns the horizontal shift between the two ir cameras, and not the shift between the depth camera and the color camera.

Unfortunately I have no way of knowing, as "QueryDSDisparityShift" will only return a value of 0.

I also thought that maybe this is tied to "SetDSEnableDisparityOutput". However when I try to enable the "Disparity Output Mode" it returns "PXCM_STATUS_DEVICE_FAILED". Other than a brief description in the documentation, I cannot find any more information on it's intended use or how to properly enable it.

I realize this is not easy stuff and that there's a lot of work left to be done. I'd greatly appreciate any help or pointers. Thanks!

[SMali10]

Sorry to bump this thread, but it seems that these functions are either broken, or not yet implemented for the R200.

Can someone from intel please let me know if this is a bug, if these functions have yet to be implemented, or if they are not meant for the R200?

Thanks!