topic Re: Depth map of black pixels in an image is not coming in Items with no label

Depth map of black pixels in an image is not coming

KKedi — Thu, 14 Sep 2017 08:50:55 GMT

Camera Model: SR300

Firmware Version: firmware': '3.21.0.0'

Operating System & Version: macOS sierra 10.12.6

librealsense version: 1.12.1

pyrealsense version: 2.1

Required info:

I am trying to get the depth map using intel realsense SR300. However i am not getting the depth map for the black pixels in an image. example: for my hair i am getting 0 value in the depth map.

I am wearing a stripes t-shirt having grey and red colour. Depth map for red stripe is generated however for the grey colour the depth map is not created. How to solve this problem? Even the depth of my hair, beard or anything black is not coming.

CODE:

import os

import sys

import cv2

import pyrealsense as pyrs

import numpy as np

def create_directory(directory):

if not os.path.exists(directory):

os.makedirs(directory)

return 1

else:

print(directory, '- already exists')

return 0

def check_directories():

try:

create_directory('./data')

create_directory('./data/depth')

create_directory('./data/color')

create_directory('./data/cad')

create_directory('./data/dac')

except:

print("Unexpected error:", sys.exc_info()[0])

return -1

return 0

def main():

file_structure = check_directories()

if file_structure == -1:

print('\nERROR: Directories can\'t be created, error thrown')

return -1

else:

print('\nDirectories created successfully...\nLaunching camera module...')

# Fire camera & launch streams

serv = pyrs.Service()

serv.start()

cam = serv.Device(device_id = 0, streams = [pyrs.stream.ColorStream(fps=10),

pyrs.stream.DepthStream(fps=10),

pyrs.stream.CADStream(fps=10),

pyrs.stream.DACStream(fps=10),

pyrs.stream.InfraredStream(fps=10),

pyrs.stream.PointStream(fps=10)])

scale = cam.depth_scale * 1000

print "scale = ", scale

# Some important variables

flag_save_frames = True

file_num = 0

# Start fetching Buffer

print('Starting Buffer...')

while(True):

cam.wait_for_frames()

current_color = cam.color[:,:,::-1]

# print current_color.shape

current_depth = cam.depth * scale

current_cad = cam.cad[:,:,::-1]

current_dac = cam.dac * scale

print "max depth", np.max(current_dac)

print "print depth val = ",np.max(cam.dac)

cv2.imshow('Color',current_color)

cv2.imshow('Depth',current_depth/1000)

cv2.imshow('CAD',current_cad)

cv2.imshow('DAC',current_dac/1000)

if flag_save_frames:

num = format(file_num, '08')

cv2.imwrite('./data/depth/' + str(num) + '.png', cam.depth)

cv2.imwrite('./data/color/' + str(num) + '.png', current_color)

cv2.imwrite('./data/dac/' + str(num) + '.png', cam.dac)

cv2.imwrite('./data/cad/' + str(num) + '.png', current_cad)

file_num += 1

k = cv2.waitKey(1)

if k == ord('q'):

print('Q Pressed...\nEnding execution')

break

if k == ord('f'):

if flag_save_frames:

print('F Pressed...\nStopped fetching frames...')

flag_save_frames = False

else:

print('F Pressed...\nStarted fetching frames...')

flag_save_frames = True

XXXWHITESPA...

Re: Depth map of black pixels in an image is not coming

MartyG — Thu, 14 Sep 2017 12:05:35 GMT

Causes of an image being black like this can include:

1. The light source in the room is too dark or too bright. The camera has a component called an IR emitter that assists the IR sensor with exposure, and if the lighting is too low or too high, the emitter can cause the IR sensor to become saturated. The IR emitter can be turned off with scripting in Librealsense.

* Enables / disables R200 emitter

public void setEmitterEnabled(int value) {

setOption(RealSense.RS_OPTION_R200_EMITTER_ENABLED, value);

}

2. The 'gain' value may be too low. Increasing gain can make an image brighter.

Also, if you have a problem with depth scans that are too noisy, using a 'point cloud' type of depth scan may yield better results.

Re: Depth map of black pixels in an image is not coming

KKedi — Fri, 15 Sep 2017 05:29:51 GMT

MartyG the light source is good enough. It is always the case that the depth for the objects black in colour is undefined. Are black objects a limitation for SR300?

The depth images are very noisy and so are there any predefined methods that helps us reduce noise?

Re: Depth map of black pixels in an image is not coming

MartyG — Fri, 15 Sep 2017 05:53:56 GMT

Further research revealed that it is apparently a fundamental principle of 3D scanning in general (not just RealSense) that black / dark objects do not scan well because the color absorbs the laser. A tip that was shared many times for scanning black objects was to cover them in a spray or a powder fine enough to spray on, such as foot powder or baby powder so that the laser can see the surface.

I recently found a link to a professional 3D scanning spray.

https://3dscanspray.com/ 3DScanSpray.com

In your case, I think I would think that coating your t-shirt in baby powder would be a nicer option than foot powder!

As I mentioned above, doing a 'point cloud' depth scan may provide better results if you have noise on your scan.

https://software.intel.com/en-us/articles/intel-realsense-technology-and-the-point-cloud Intel® RealSense™ Technology and the Point Cloud | Intel® Software

Re: Depth map of black pixels in an image is not coming

Michael_M_Intel1 — Fri, 15 Sep 2017 06:06:33 GMT

As I noted before, shiny and/or transparent objects have a similar problem (in this case, the light is not absorbed, but reflected or refracted in a direction not visible to the camera). Hair is yet ANOTHER problem: it's just too fine to reflect the dot pattern in a coherent way, and often individual hairs are below the camera resolution. Of course if the hair is black you have more than one problem. Edges of objects are also often not picked up well as they are at an extreme angle with respect to the camera (and may be shadowed from one camera or the other), and also pick up relatively small amounts of light from the projector. As an added complication, objects that are "glossy" tend to act more and more "shiny" (more specular, or mirror-like) at glancing angles.

These are not RealSense problems per se but general problems with depth sensing.

Unfortunately using a spray or otherwise modifying the object to be scanned (eg using a hair cap) is not always feasible. Your best strategy is probably to make your algorithms robust to holes in the data, and using hole-filling algorithms were appropriate. Don't forget you also have color pixels even where there is no depth so you can peek at them to see if a hole is a "real" hole or maybe due to the pixel being dark (which you can easily determine from the color channel).

Re: Depth map of black pixels in an image is not coming

KKedi — Fri, 15 Sep 2017 09:30:04 GMT

Hi MartyG Thanks for your response.

R200 camera uses a similar technology. However, such a problem is not present there. I had my friend using it and he was able to get the depth maps for the black objects as well.

Re: Depth map of black pixels in an image is not coming

MartyG — Fri, 15 Sep 2017 09:55:00 GMT

I did further research into scanning black objects, and every information source repeated the earlier suggestions of changing the lighting conditions or using a spray / powder. The only new tip I did see suggested was to turn the camera 90 degrees to change how the laser light falls onto the object being scanned.

Re: Depth map of black pixels in an image is not coming

PPola — Tue, 10 Oct 2017 10:08:07 GMT

Do you know if new generation of realsense D415 & D335 deal better with this problem?

Re: Depth map of black pixels in an image is not coming

MartyG — Tue, 10 Oct 2017 10:19:20 GMT

Whilst the new cameras would still be subject to the same general physics problem of dark / black colors absorbing laser light, they do have a new Advanced Mode that enables users to have greater control over depth generation and color correction.

https://github.com/IntelRealSense/librealsense/blob/development/doc/rs400/rs400_advanced_mode.md librealsense/rs400_advanced_mode.md at development · IntelRealSense/librealsense · GitHub