How can I get the Depth and Color image registered with the Unity wrapper? Also, is there any Guide I can follow?

[AJuan2]

I've made a function in a .dll that registers 3 images together (color, depth, thermo), for that I need 3 IntPtr.

I need to make a UI useing Unity to show the registered images, so I'm using the RealSense Unity wrapper to pick up the color and depth images from the stream.

I'm following the "AligmentSample" from the realsense.unitypackage but I can't find where I can get both images registered (Color and Depth) at the same time.

I've tryed to get them from the "OnNewSample(Frame frame)" function in "RsStreamTextureRender.cs" but the problem is that on the first call of the function I get the color ant in the second de depth, but I need to get both at the same time.

Does anyone know how I can get Depth and Color frames at the same time?

If not, is there any Guide apart from the github readme?

I'll really appreciate any help.

Thanks!

[MartyG]

Apparently, 'registration' means the same as 'alignment' in terms of the RealSense SDK. So you may find examples more easily if you search by the keyword 'align' instead of 'register'.

Another user had problems with depth and color alignment in Unity. They were advised to avoid creating an Align object for every frame.

https://github.com/IntelRealSense/librealsense/issues/3574

I see that you have also posted your question on the GitHub. That is likely to be the best place for Unity programming advice.

I do not know of another good source for Unity Librealsense documentation other than the GitHub, unfortunately.

[AJuan2]

Hey Marty thanks for that quick response!!

I've tryed that example and it kind of worked, now I get the frames registered but not all of them.

The color image streams like 15 continuous frames and then the 15th frame frozes getting 72 frames of that same frame, that pattern persists during the whole run time. Ex. (0 1 2 ... 13 14 15 15 15 15 ... 88 89 90 ... 102 102 102 ...)

While the depth image stream has no problem.

Here is my code:

/// <summary> /// Worker Thread for multithreaded operations /// </summary> Align align = new Align(Stream.Color); CSRegister.CSRegisterTCD regClass = new CSRegister.CSRegisterTCD("C:\\Users\\SomePath\\spaceing24_extrinsics_640_80.xml", true); //FrameSet fs; int counter = 0; private void WaitForFrames() { while (!stopEvent.WaitOne(0)) { using (var frames = m_pipeline.WaitForFrames()) { Frame aligned_frames = align.Process(frames); RaiseSampleEvent(aligned_frames);   // Write color and depth to the disk regClass.writeRGBimage(frames.ColorFrame.Data, "imageTest/col_" + (counter) + ".png", 640, 480); regClass.writeGRAYimage(aligned_frames.Data, "imageTest/dep_" + (counter++) + ".png", 640, 480); aligned_frames.Dispose(); } } }

The functions "writeRGBimage(IntPtr, String, int, int)" and "writeGRAYimage(IntPtr, String, int, int)" takes the images and writes them to the disk.

I've also tryed to get the color image with "aligned_frames.As<FrameSet>().ColorImage.Data" but the same happends.

Any Idea how to get continuous stream without problems?

That peace of code is from the "RsDevice.cs" function btw.

Thanks for your help!

[MartyG]

I looked over the code very carefully and checked it against other reference sources. I wonder if the problem is occurring in the using statements, and that the color hasn't got a using statement defined for it. The C# example script in the link below for getting color and depth frames may illustrate what I mean.

https://github.com/IntelRealSense/librealsense/issues/3413#issuecomment-474451376

[AJuan2]

That was it, If I put the code inside the using statements works no problem.

Thanks for replying!

This is how I did it if anyone wonders:

/// <summary> /// Worker Thread for multithreaded operations /// </summary> Align align = new Align(Stream.Color); CSRegister.CSRegisterTCD regClass = new CSRegister.CSRegisterTCD( "E:\\somepath\\spaceing24_extrinsics_640_80.xml", true); int counter = 0; FrameQueue qColor; FrameQueue qDepth; IntPtr thermo, thermoOverColor; int w, h; private void WaitForFrames() { while (!stopEvent.WaitOne(0)) { using (var frames = m_pipeline.WaitForFrames()) using (Frame aligned_frames = align.Process(frames)) if (aligned_frames.IsComposite) { using (FrameSet fs = aligned_frames.As<FrameSet>()) using (Frame color = fs.ColorFrame) using (Frame depth = fs.DepthFrame) { Marshal.FreeHGlobal(thermo); thermo = regClass.readGRAYimage("E:\\path\\t_2019711561978742124.png"); regClass.writeGRAYimage(depth.Data, "imageTest/depth_" + (counter) + ".png", 640, 480); regClass.writeRGBimage(color.Data, "imageTest/color_" + (counter) + ".png", 640, 480);   if (isRegisterTCD) { Marshal.FreeHGlobal(thermoOverColor); regClass.update(thermo, color.Data, depth.Data, true, 0.0f, 1.0f, 0.5f, out w, out h); thermoOverColor = regClass.getThermoOverColor(); regClass.writeRGBimage(thermoOverColor, "imageTest/result_" + (counter) + ".png", w, h); Debug.Log("REGISTER_" + counter); } counter++; RaiseSampleEvent(aligned_frames);   } } } }

Now my problem is that I have the RGB image "thermoOverColor" (an IntPtr) that I need to show in the UI in sync with the color and depth. I don't know how but that's another different question.

Really appriciate your help Marty.

Thanks!

[MartyG]

Thanks so much for sharing your solution!​

Apologies for the delay in responding, as this forum does not send ​me notification when new comments are posted. So it is hard to track updates on older discussions.

A simple way to show live RGB would be to create a display surface in your UI and use Unity's webcamtexture instruction to stream RGB data onto that surface.

https://docs.unity3d.com/ScriptReference/WebCamTexture.html

[AJuan2]

Sorry for the late response!

I'm still trying to show that IntPtr (an RGB image) into a canvas, the problem is that it isn't a camera stream but the result of the registration of the thermal camera with the color frame, so the WebCamTexture object has no device to conect to.

So now I'm trying to use Texture2D loading the pointer with tex.LoadRawTextureData(thermoOverColor, w * h) but this error shows up:

LoadRawTextureData_ImplPointer can only be called from the main thread.

Constructors and field initializers will be executed from the loading thread when loading a scene.

Don't use this function in the constructor or field initializers, instead move initialization code to the Awake or Start function.

UnityEngine.Texture2D:LoadRawTextureData(IntPtr, Int32)

RsDevice:WaitForFrames() (at Assets/RealSenseSDK2.0/Scripts/RsDevice.cs:208)

Even if I initialize the variable in the Awake function, still shows up.

So I've tryed to debug but now in the line 16 of the example before shows the following error:

System.Exception: Frame didn't arrived within 5000 ---> System.Runtime.InteropServices.ExternalException: rs2_pipeline_wait_for_frames(pipe:0000000043923A20)

 --- End of inner exception stack trace ---

 at Intel.RealSense.ErrorMarshaler.MarshalNativeToManaged (IntPtr pNativeData) [0x00000] in <filename unknown>:0

 at (wrapper managed-to-native) Intel.RealSense.NativeMethods:rs2_pipeline_wait_for_frames (intptr,uint,object&)

 at Intel.RealSense.Pipeline.WaitForFrames (UInt32 timeout_ms) [0x00000] in <filename unknown>:0

 at RsDevice.WaitForFrames () [0x00012] in C:\Users\path\RsDevice.cs:189 happends

And it goes out if I erase the code after the line 16 in the example.

Any Idea why I can't debug the code with the example before? or there is any way to increase the waiting frame time?

Thanks for all your help!

[MartyG]

At this point we are going into the workings of the RealSense Unity wrapper at a complexity that I am unfamiliar with. The best place to get an answer to this particular question will be at the RealSense GitHub, as that is where the developers who actually write the Unity wrapper reside. I'm sorry I couldn't be of more help on this aspect.

To post at the GitHub site, please visit the link below and click the 'New Issue' button.

https://github.com/IntelRealSense/librealsense/issues

[AJuan2]

No problem! I'll ask on github!

Thanks for all your help Marty!