Items with no label
3335 Discussions

Realsense 2.0 D415 - Depth to Terrain - Unity c#

atiwa11
Beginner
1,618 Views

I am trying to generate a runtime terrain from the depth values of the realsense camera D415.

I just got the camera to try and do something for the summer. never used more than webcams in unity before so im very lost even after going through all the sample scenes.

 

my terrain is 500 x 500 x 40

using realsense sdk 2.0

 

Here is the code below. IF i run this unity goes full silent treatment on my incompetence im very sure im doing something gravely wrong. any help is greatly appreciated. Right now the code works with Perlin Noise. I just need to find out how to get the depth values (z values) for the xand y position.

using UnityEngine; using Intel.RealSense; public class TerrainGenerator : MonoBehaviour { public int depth = 40; public int width = 500; public int height = 500; public float scale = 20f; public float xOffset = 10; public float yOffset = 10; private DepthFrame depthFrame; public bool scroll; public float scrollSpeed; private Pipeline pipe; private void Start() { /// pipe = new Pipeline(); pipe.Start(); /// xOffset = Random.Range(0f, 9999f); yOffset = Random.Range(0f, 9999f); scroll = false; scrollSpeed = 0; } void Update() { Terrain terrain = GetComponent<Terrain>(); terrain.terrainData = GenerateTerrain(terrain.terrainData); if(scroll) xOffset += Time.deltaTime * scrollSpeed; } TerrainData GenerateTerrain(TerrainData tData) { tData.heightmapResolution = width + 1; tData.size = new Vector3(width, depth, height); tData.SetHeights(0, 0, GenerateHeights()); return tData; } float[,] GenerateHeights() { float[,] heights = new float[width, height]; for (int x = 0; x < width; x++) { for (int y = 0; y < height; y++) { //heights[x, y] = CalculateHeight(x, y); var frames = pipe.WaitForFrames(); var depth = frames.DepthFrame; heights[x, y] = depth.GetDistance(x, y); } } return heights; } float CalculateHeight(int x, int y) { //float xCoord = (float)x / width * scale + xOffset; //float yCoord = (float)y / height * scale + yOffset; //return depthFrame.GetDistance(x,y); //return Mathf.PerlinNoise(xCoord, yCoord); using (var frames = pipe.WaitForFrames()) using (var depth = frames.DepthFrame) return depth.GetDistance(x, y); } }

 

0 Kudos
4 Replies
MartyG
Honored Contributor III
1,151 Views

Several years ago there was a Unity-made RealSense game called 'Tanked!' that looked down at a sandpit from above and read the depth contours of the sand in realtime as the players moved the sasnd with their hands. The creators did a detailed making of article. It is for the older 2016 RealSense SDK but the code listings may give you some ideas.

http://hyd23.rssing.com/chan-12487997/all_p99.html

0 Kudos
atiwa11
Beginner
1,151 Views

thank you Marty for the reply.

From what i understand through the code snipet in the article, the sdk changed a lot from using PXCM to using pipeline RS now. I will try and contact DesignMill to see if they can give me some advise on how to go about it.

 

again thank you for the lead. If you can think of any other examples to pass along my way i would really appreciate it.

seems like no one has actually documented the use case of depth mapping using the 2.0 sdk.

 

thanks,

Anurag

0 Kudos
MartyG
Honored Contributor III
1,151 Views

I found a case where a RealSense user generated a depth map in Unity and RealSense SDK 2.0 with C#.

 

https://github.com/IntelRealSense/librealsense/issues/3134

0 Kudos
atiwa11
Beginner
1,151 Views

sorry for the late response ( hurricane season on the east cost )

Thanks for following up.

All these solutions im trying make Unity stop responding on pressing the play button or nothing happens.

I get a blank depth array that is 640*480*9 in length. I dont understand where this big an array is coming from if the depth is supposed to be 640*480 resolution with depth value for each pixel.

I just cant map it to a an array.

after googling a bit i found out that everytime someone asks this, (forum, github issue...) they either magically solve it and are never to be seen again on the the forum or it doesnt get answered properly with people guiding the poster towards c# cookbook

the depth cutoff script itself which ships with the package is not ever used in the samples in the sdk.

 

Again thank you for following up. I will keep researching this and definitely will post a solution if i find one. If you have any more leads or any contacts that i can get in touch with who can give me unity specific guidance then that will be grand!

I need to be able to stream the depth data and not just take a 1 frame snapshot. the point cloud data being in ushort format in that massive array really makes it a pain.

 

this is how far i have gotten: I have created a terrain which is 640x480. I followed the cookbook and stuff from the links you graciously provided.

but the error index out of range is obvious here since the 1D depth array is 2764800 items longs which is 640*480(resolution of the depth frame) * 9 (what is this 9) ?

 

 

Here is the code.

 

using UnityEngine; using Intel.RealSense;     public class TerrainGenerator : MonoBehaviour {   public int depth = 40; public int width = 1920 ; public int height = 1440;   public float scale = 20f;   public float xOffset = 10; public float yOffset = 10;     public bool scroll; public float scrollSpeed;     private float[] vertices; private Pipeline pipe; private PointCloud pc;     private void Update() { /* //realsense 1d array pipe = new Pipeline(); pc = new PointCloud();   pipe.Start();   //Debug.Log(vertices.Length + " : is the length of the array");       xOffset = Random.Range(0f, 9999f); yOffset = Random.Range(0f, 9999f);   scroll = false; scrollSpeed = 0; */   Terrain terrain = GetComponent<Terrain>(); terrain.terrainData = GenerateTerrain(terrain.terrainData);       } void LateUpdate() { /* using (var frames = pipe.WaitForFrames()) using (var depth = frames.DepthFrame) using (var points = pc.Process(depth).As<Points>()) { // CopyVertices is extensible, any of these will do: vertices = new float[points.Count * 3]; // var vertices = new Intel.RealSense.Math.Vertex[points.Count]; // var vertices = new UnityEngine.Vector3[points.Count]; // var vertices = new System.Numerics.Vector3[points.Count]; // SIMD // var vertices = new GlmSharp.vec3[points.Count]; // var vertices = new byte[points.Count * 3 * sizeof(float)]; points.CopyVertices(vertices);     } */   if (scroll) xOffset += Time.deltaTime * scrollSpeed; } TerrainData GenerateTerrain(TerrainData tData) { tData.heightmapResolution = width + 1; tData.size = new Vector3(width, depth, height); tData.SetHeights(0, 0, GenerateHeights()); //tData.SetHeights(0, 0, Convert1DTo2DArray()); return tData; } float[,] GenerateHeights() { float[,] heights = new float[width, height]; for (int x = 0; x < width; x++) { for (int y = 0; y < height; y++) { heights[x, y] = CalculateHeight(x, y); //later will become the depth value from the real sense camera } } return heights; }   float CalculateHeight(int x, int y) { float xCoord = (float)x / width * scale + xOffset; float yCoord = (float)y / height * scale + yOffset; return Mathf.PerlinNoise(xCoord, yCoord); } /* float[,] Convert1DTo2DArray() { int width = 1920; int height = 1440; var Map = new float[width, height]; for (int i = 0; i < width; i++) { for (int j = 0; j < height; j++) { Map[j, i] = vertices[j * 1920 + i] * 100; } } return Map; } */   }

 

0 Kudos
Reply