Outside/sunlight functionality and Linux support

[Charles_W_]

Hello,

I'm a graduate student at NC State University.  My research group is working on a project for the North Carolina Department of Transportation to use Quadrotor UAVs to assist in 3D mapping.  As part of that project, we are trying to determine the best way to construct a 3D model of a bridge after flying around it.

The Realsense sensor seems perfect for a SLAM application like this (and ground robot mapping/navigation).  The key question is whether or not the sensor can work outside in direct sunlight.  Some things I have read claim the sensor is purely sterio vision based and can function perfectly outside.  Others have claimed that it is a structure light sensor (like its predecessor, the Creative Senz3D sensor) and will have the same difficulties as sensors such as the Kinect or Primesense Capri.  So...  If I may ask, does the Realsense sensor work outside, pointed at objects in direct sunlight?

My other question relates to library support.  I understand that the sensor currently supports only Windows 8.  There is however, efforts to port it to Android (presumably to go on some of the tablets).  If I may ask, since Android is based on top of Linux, is there likely to be Linux support any time soon?  It would be very handy if there was, as I believe the majority of mobile robots are Linux based (and Willow Garage's ROS is primarily Linux based).

Thanks,

Charlie West

[Colleen_C_Intel]

One other "limitation" to consider is the range of the camera. The current front-facing Windows Real Sense cam is limited to just over 1 meter so probably not sufficient. There are plans for additional rear facing cameras in multiple OS but I don't know the range of those at this time.

[Andradige_S_Intel]

The current version of the SDK only supports Windows 8 and the compatible 3D camera is intended for smaller depths in the range of 1-2 meters.

[Charles_W_]

Thank you for taking the time to address my comment.

You have both raised legitament concerns, but the original questions have still not been answered. 

If I may ask, will the Intel Realsense 3D work in direct sunlight?

Also, is Linux support likely to happen in the next two years?

Thanks,

Charlie

[Andradige_S_Intel]

Charles,

Sorry but I cannot comment on non-public contents including road maps.

[Alex_K_2]

Hello Charles! Just FYI, it's public data: Windows today….Android on the way Look for more details on Intel® RealSense™ SDK for Android later this year From Intel presentation: http://www.iai.uni-bonn.de/~gall/tutorials/slides14/ho_cvpr2014_rgbd_tutorial.pptx P.S. Intel guys need to be more friendly for future developers :)

[Tim_T_]

I've received and tried the R200 outdoors in sunlight, and it doesn't seem to work at all in sunlight.  In outdoors areas that are suitably shaded, it works.  I think any claim that the R200 works outdoors is misleading.  If anyone has different experience with using the R200 outdoors, I'd love to hear about it.

     ...Tim...

[samontab]

Since the F200 and R200 work by reading a specially projected IR light, they can't compete with the overwhelming amount of IR light the sun provides. The sensor just saturates.

[SMali10]

@samontab

I believe the R200 is designed for outdoor use, it just hasn't been implemented in the SDK yet.

Both here: https://software.intel.com/en-us/articles/realsense-r200-camera

and here: https://software.intel.com/en-us/realsense/devkit

They mention the R200 working at longer ranges outdoors. In the current release of the SDK there is something called "disparity mode" which can be enabled but currently does not do anything. Also the gain of the infrared cameras are set extremely high and cannot be manually changed like the exposure time or framerate.

***Everything below this line is conjecture and has not been officially confirmed by intel***

From what I can tell the R200 does not currently use the right IR camera for estimating depth. It uses the IR projector and the left IR camera, similar to how the V1 Kinect, primesense cameras, and most other laser projection depth cameras calculate depth.

My guess is when taken outside the R200 will enter "disparity mode". The IR projector is disabled, and the IR cameras expose for the scene using sunlight. Depth will then be measured using the disparity between the two IR cameras.

This is much more processor intensive than the current structured light approach, especially in realtime, so it's probably still under heavy development for a future release.

In theory, you could create a program that turns off the IR projector, set the exposure time of the IR cameras extremely low, and add a neutral density filter to the IR cameras. This should get you some kind of usable exposure outside, or at least in indirect sunlight. You can then use OpenCV to calibrate the IR cameras, and create a depth map using the stereo disparity functions (technically the IR cameras have already been calibrated and rectified, however the functions for getting the calibration data are all broken).

[samontab]

Hi Sam,

I don't have an R200, so I can only speculate as well. It seems to me that the R200 is based on active IR stereo.

If this is true, then the projected IR dots would appear at different locations on each IR camera (left/right), so you could calculate disparity, and then distance from this. This is great because you have a reliable stereo system that works even on surfaces with no texture (i.e. a white wall), and also works at night.

Now, the problem with this approach is that the sun is much more powerful than the IR emitter, so the sensor will get saturated.

Doing what you propose basically transforms the RealSense camera into a standard stereo system, just in IR instead of visible. This has the common problem of not being able to infer depth from featureless areas, such as a white wall. And this case may be even trickier because it is not obvious what regions will be featureless in IR, as opposed to visible because we can just see them.

But, if you need to use it outdoors, it is probably the best workaround for this particular sensor.

[SMali10]

Actually I was wrong, the depth stream doesn't work if either of the ir cameras are blocked. However it seems to only be gathering depth from the projected light pattern, probably because it's easier to identify.

Like I mentioned the ir emitter would probably be disabled in an "outdoor mode", and the exposure of the ir cameras would be adjusted (right now the gain is locked very high). Then it would behave like a standard stereo system based on feature detection.

The light spectrum of the R200 ir cameras seem to be very broad, especially compared to other ir cameras. It doesn't capture all light, but I have gotten illumination off of some indoor/incandescent lights.

There is no magic bullet for depth, but projected using ir in low light and stereo feature detection in sunlight seems to be the best of both.

Intel has basically confirmed that the R200 will work outdoors (it's in the specifications on the sales page). They just haven't confirmed the how yet.