How to extract the depth information of the face and save it in the .ply file ?

[wshao4]

I encountered a problem, trying to extract the depth information of the face and save it in the .ply file. But the report was wrong. See here for details.

[Eliza_D_Intel]

Hello wshao4, Thank you for your interest in the Intel RealSense Products. We are looking into our request and we will reach you as soon as possible. Thank you, Eliza

[JesusG_Intel]

Hello wshao4,

Attached is a python script that will help. It does not work perfectly because we are not experts on Open3D but it should take you in the right direction. The attached code joins your code with this one: https://github.com/IntelRealSense/librealsense/issues/2769. Note that we modified this line:

color_interest = color_image[y:h, x:w,:].reshape(-1,3).

import pyrealsense2 as rs import numpy as np import cv2 import dlib import os from open3d import * # Configure depth and color streams pipeline = rs.pipeline() config = rs.config() config.enable_stream(rs.stream.depth, 640, 480, rs.format.z16, 30) config.enable_stream(rs.stream.color, 640, 480, rs.format.bgr8, 30) # Declare pointcloud object, for calculating pointclouds and texture mappings pc = rs.pointcloud() # Start streaming pipeline.start(config) align_to = rs.stream.color align = rs.align(align_to) count = 0 try: while count < 20000: count +=1 # Wait for a coherent pair of frames: depth and color frames = pipeline.wait_for_frames() depth_frame = frames.get_depth_frame() color_frame = frames.get_color_frame() if not depth_frame or not color_frame: continue # Convert images to numpy arrays depth_image = np.asanyarray(depth_frame.get_data()) color_image = np.asanyarray(color_frame.get_data()) # Detect face frame_new = cv2.cvtColor(color_image, cv2.COLOR_BGR2RGB) hogFaceDetector = dlib.get_frontal_face_detector() faceRects = hogFaceDetector(frame_new) for det in faceRects : x = det.left() y = det.top() w = det.right() h = det.bottom() # [det.left():det.top(),det.right():det.bottom()] # Apply colormap on depth image (image must be converted to 8-bit per pixel first) depth_colormap = cv2.applyColorMap(cv2.convertScaleAbs(depth_image, alpha=0.03), cv2.COLORMAP_JET) # Stack both images horizontally images = np.hstack((color_image, depth_colormap)) # Show images cv2.namedWindow('RealSense') cv2.rectangle(images,(x,y),(w, h),(255,0,0),2) cv2.imshow('RealSense', images) # Tell pointcloud object to map to this color frame pc.map_to(color_frame) # Generate the pointcloud and texture mappings # Align the depth frame to color frame aligned_frames = align.process(frames) # Get aligned frames aligned_depth_frame = aligned_frames.get_depth_frame() aligned_color_frame = aligned_frames.get_color_frame() points = pc.calculate(depth_frame) vertices = np.asanyarray(points.get_vertices(dims=2)) width = aligned_depth_frame.get_width() image_Points = np.reshape(vertices , (-1,width,3)) if not aligned_depth_frame or not aligned_color_frame: continue vertices_interest = image_Points[y:h, x:w,:].reshape(-1,3) color_interest = color_image[y:h, x:w,:].reshape(-1,3) pcd = PointCloud() pcd.points = Vector3dVector(vertices_interest) pcd.colors = Vector3dVector(color_interest) draw_geometries([pcd]) write_point_cloud(str(count)+'.ply', pcd) if cv2.waitKey(1) & 0xFF == ord('q'): break finally: # Stop streaming pipeline.stop()

The RealSense SDK portions are done the right way in the code, you just need to figure out the Open3D PLY export part.

Regards,

Jesus G.

Intel Customer Support

[wshao4]

Thanks , I had tried to export by using open3D using the sunit1409's code like yours, but get the .ply file which don't contain the element face. the details is at the end of here.

[JesusG_Intel]

Hello wshao4, in your post on Github you said that the second file that was generated with the SR300 does have the facial elements? Please confirm. Your code works well with the SR300? Which D400 camera are you using? We forgot to ask this very basic question. Regards, Jesus Intel Customer Support

[wshao4]

Yes，my code works well with the SR300. but can't get the face elements. ​I also try your code just now , get the same results which not containing face elements.

[JesusG_Intel]

Can you send me your code so I can test it? Are you using only the SR300 or are you also using a D400 series camera? You can get the face elements with the SR300?

[wshao4]

I only used the SR300​, I can't get the face elements. The following is my code, it's very similar to yours.

import pyrealsense2 as rs import numpy as np import cv2 import dlib import os from open3d import *     # Configure depth and color streams pipeline = rs.pipeline() config = rs.config() config.enable_stream(rs.stream.depth, 640, 480, rs.format.z16, 30) config.enable_stream(rs.stream.color, 640, 480, rs.format.bgr8, 30)   # Declare pointcloud object, for calculating pointclouds and texture mappings pc = rs.pointcloud()   # We want the points object to be persistent so we can display the last cloud when a frame drops points = rs.points()   # Start streaming profile = pipeline.start(config)   # Open3D align_to = rs.stream.color align = rs.align(align_to) count = 0     try: while count < 20000 : count += 1 # Wait for a coherent pair of frames: depth and color frames = pipeline.wait_for_frames() # Align the depth frame to color frame aligned_frames = align.process(frames) # Get aligned frames depth_frame = aligned_frames.get_depth_frame() color_frame = aligned_frames.get_color_frame() # Generate the pointcloud and texture mappings points = pc.calculate(depth_frame) vertices = np.asanyarray(points.get_vertices(dims=2)) w = depth_frame.get_width() image_Points = np.reshape(vertices , (-1,w,3)) if not depth_frame or not color_frame: continue color_image = np.asanyarray(color_frame.get_data()) # Detect face frame_new = cv2.cvtColor(color_image, cv2.COLOR_BGR2RGB) hogFaceDetector = dlib.get_frontal_face_detector() faceRects = hogFaceDetector(frame_new) for det in faceRects : x = det.left() y = det.top() w = det.right() h = det.bottom() print(x,w,y,h) color_image = np.asanyarray(color_frame.get_data())   #if len(depth_image) <= 0: # continue print(x,y,w,h) vertices_interest = image_Points[y:h, x:w,:].reshape(-1,3) color_interest = color_image[y:h, x:w,:].reshape(-1,3)   pcd = PointCloud() pcd.points = Vector3dVector(vertices_interest) pcd.colors = Vector3dVector(color_interest) draw_geometries([pcd])   write_point_cloud('2.ply', pcd) # Apply colormap on depth image (image must be converted to 8-bit per pixel first) # depth_colormap = cv2.applyColorMap(cv2.convertScaleAbs(depth_image, alpha=0.03), cv2.COLORMAP_JET)   # Stack both images horizontally #images = np.hstack((color_image, depth_colormap))   # Show images #cv2.namedWindow('RealSense') #cv2.rectangle(images,(x,y),(w, h),(255,0,0),2) #cv2.imshow('RealSense', images) cv2.waitKey(1)   finally:   # Stop streaming pipeline.stop()

[wshao4]

@Jesus G.​ , Do you have the solutions getting the face elements now ? it is very import to me. Can any one help me, thanks.

[JesusG_Intel]

Hello @wshao4​, I ran your code and was able to get the face elements using the SR300 camera. See the attached files. As you can see in the PLY, there are 32220 vertices.

Regards,

Jesus G.

Intel Customer Support

[wshao4]

The "2.ply" don't contain face elements.​😨 You can have a closer look. 😂

[JesusG_Intel]

Please clarify what facial elements you are looking for. You can see in the JPG the shape of the face created by all of the vertices.

[wshao4]

The following contents contain face elements which is created by intel sdk. You can see, there have 107088 face elements

ply format binary_little_endian 1.0 comment pointcloud saved from Realsense Viewer element vertex 55663 property float32 x property float32 y property float32 z property uchar red property uchar green property uchar blue element face 107088 property list uchar int vertex_indices end_header

​While the folloing contents don't contain face elements, which created by open3D. of course, it had extracted only the roi information of the face. It don't have face elements, only have vertex element.

ply format binary_little_endian 1.0 comment Created by Open3D element vertex 32220 property double x property double y property double z property uchar red property uchar green property uchar blue end_header

[wshao4]

@Jesus G.​ The following content is saved from Realsense Viewer，which have 107088 face elements and 55663 vertex element

ply format binary_little_endian 1.0 comment pointcloud saved from Realsense Viewer element vertex 55663 property float32 x property float32 y property float32 z property uchar red property uchar green property uchar blue element face 107088 property list uchar int vertex_indices end_header

​While, The following content is created by open3D, which don't have face elements, only have 32220 vertex element. Also it had extracted the roi of face.

ply format binary_little_endian 1.0 comment Created by Open3D element vertex 32220 property double x property double y property double z property uchar red property uchar green property uchar blue end_header

[wshao4]

@Jesus G.​ Hi, the following content is saved from Realsense Viewer，which contain element vertex 55663 and element face 107088

ply format binary_little_endian 1.0 comment pointcloud saved from Realsense Viewer element vertex 55663 property float32 x property float32 y property float32 z property uchar red property uchar green property uchar blue element face 107088 property list uchar int vertex_indices end_header

​While, The other following content is created by Open3D, which only contain element vertex 32220

ply format binary_little_endian 1.0 comment Created by Open3D element vertex 32220 property double x property double y property double z property uchar red property uchar green property uchar blue end_header

[wshao4]

Sorry, The page did not show my reply, I thought that the reply was not successful.Reply to you twice more @Jesus G.​ 

[JesusG_Intel]

Hello, I see what is going on. The Pointcloud geometry in Open3D does not contain a "faces" element. It has points, colors, and normals. The TriangleMesh geometry has vertices and triangles and colors. Unfortunately, Open3D does not have a method for converting pointclouds to meshes at the moment. Librealsense creates the face elements programatically in the function points.export_to_ply as can be seen in src/archive.cpp line 48. You will have to find a way to get only the points you want from the points class (points = pc.calculate(depth_frame) and then use points.export_to_ply to get a ply file with face elements. This is beyond the scope of our technical support function.