RealSense versus SteamVR / HTC Vive - a comparison

[MartyG]

Hi everyone,

I've been studying several detailed articles on the mechanics of Valve and HTC's SteamVR / Vive motion-control and VR headset system, which has been wowing hands-on users.  My goal was to see how its principles could be replicated in RealSense (without breaking patents, of course!)  Hee are my thoughts

LASER SENSORS

Vive, via its two "Lighthouse" laser sensor boxes placed in opposite corners of a room, can accurately determine the position of the user within the room and observe what they are doing.  The lasers are attached to a cylinder in the Lighthouse sensor box that spins rapidly and casts a laser grid across the room.  It can track a number of points 100 times a second.

It would perhaps not be too difficult to mount the desktop F200 camera on a rotating device and spin it similarly to cast a grid instead of having the laser spread across a fixed area in front of it.  It is unknown though how well the tracking could be maintained under those circumstances.  There are tricks you can use to prevent the tracking from stalling though, such as putting a 'Hand Lost' or 'Face Expression Detected' rule into an object's "Start" conditions so that if tracking is lost, the camera suddenly wakes up like a tired driver who has nodded off at the wheel for a second and continues tracking.

ROOM-SCALE WALKING

One of the biggest selling points of Vive is that it supports the walking around of the user in a 15 foot room (with smaller user-customized spaces being possible in the final consumer version of Vive.)  It uses a warning system called Chaperone to display a grid overlay and emit an audible warning if the user is approaching a solid obstacle in the real world such as a wall or piece of furniture.  This is necessary because the user's view of the real world is obscured by Vive's VR headset.

It also draws a similar grid around the user's feet to warn them if they are about to kick something in the real room.

We already know that RealSense can detect knee and foot / toe joints (as it treats them like hand joints), so it is no problem for RealSense to take inputs from these joints (so long as the real-life hands are placed behind the back so the camera does not get confused between the hand and foot inputs).  These inputs could be used to make an avatar character walk, but could equally be used to  track the position of the user's feet in a room.

Detecting the real room's furniture and warning the user of it would be trickier, but perhaps feasible if the 3D Segmentation function of RealSense were used to recognize real objects in front of the camera and incorporate them into the simulation as virtual objects so that the user can be warned if their foot joints are approaching them.

There is also the potential to incorporate the real contents of the room into the simulation if RealSense were combined with a VR headset such as Oculus Rift, with the user seeing a virtual object in the headset that corresponds to the dimensions of the real object, thus allowing them to use their hands to tangibly touch and pick up that object, believing that it is an object in the simulation.

By incorporating physical stimulus into a VR simulation, you can quickly remove doubt from the user's mind about an object's true nature.  For example, if you are trying to close your eyes and imagine yourself in a magical pony castle, your mind may be skeptical about the truth of this mental image.  But if you design the terms of the visualization to fit the contents of your real environment - like laying on your real bed and imagining yourself laying *on a bed in the royal bedroom* of the magical pony castle, then the physical sensation of the real bed beneath your back will reinforce the truth of the virtual simulation to your mind and you will be more deeply immersed in it.

VIRTUAL HANDS AND HAPTIC FEEDBACK

An advantage that RealSense has over Vive is that RealSense can track individual hand joints, whilst Vive uses a handheld physical controller that resembles a grippable stick with a satellite dish on the end and buttons to perform actions such as gripping virtual objects.  The advantage of Vive's physical controller though is that it can provide haptic feedback sensations to the real hand.  

Whilst RealSense does not have such haptic feedback devices, it is possible to replicate physical sensations through careful simulation design.  When the real fingers are bent and kept in tension, the person feels that bending as a constant stimulus via their nervous system.  So if the real hand is used to grip an on-screen object with a virtual hand, the user will feel the sensation of holding that virtual object for as long as their real fingers are bent to maintain the on-screen hold on that object.

Through the psychological principles of Neuro Linguistic Programming (NLP), you can also guide the user in setting up mental programs that will subconsciously run whenever a stimulus is detected by the nervous system in a particular part of the body.  For example, if you set up a mental instruction that your body will gain energy when your middle finger is bent and then you bend that finger, the mental program will run automatically for as long as the finger is bent and the sensation of doing so is present and you believe that the instruction is working (doubt is the little sensation-killer of the mind!)  

The more often that the trigger is repeated, the stronger its effects become over time.  This phenomenon allows the simulation designer to assign an action to the nervous system as though it is a second brain, whilst having the main brain perform a different task at the same time.

Belief can be reinforced in the mind further by providing visual cues in the simulation that back up the truth of the stimuli that the user is experiencing in their body.  For example, an NLP instruction designed to tell the user's mind to raise their energy when a certain finger is bent could be accompanied by a special effect around an on-screen avatar or first-person hands, or a change in the appearance of the simulation environment such as a lighting filter that makes it look brighter.

****

I hope the above notes give you some ideas about how to take your RealSense projects to an even higher level!

[MartyG]

When thinking some more today about how the RealSense camera could respond to Neuro Linguistic Programming (NLP) gestures made by a user (see above), I reflected on another recent post in which the detection of skin color changes caused by blood flow was discussed.

https://software.intel.com/en-us/forums/topic/542819

In NLP, you can trigger programming instructions in the mind (such as feeling more confident) by training the mind to respond to a mental or physical trigger - an example of a physical sensation trigger being a tap on the shoulder or the back of the hand.

It would be easy enough to get the camera to respond to a physical gesture made with a body point that is trackable, such as the bending of a finger.  But if a person's chosen trigger involved a non-tracked part of the body, such as putting the shoulder or leg in a state of tension, how would the camera be able to recognize that?

It occurred to me that any body part that is put in tension will probably have blood pooling around the tensed area, like the hand knuckles visibly reddening when the fingers are clenched tightly into a fist.  This would show up on the camera's infra red (IR) function as an area of skin color that was subtly different to the surrounding skin.  Programming an application to respond to such a change would enable physical NLP  triggers initiated by a person to be detected and cause an effect in an on-screen object (such as an avatar body) to be activated.

Seeing a visual on-screen response to the trigger would provide reinforcement to the user's mind that the gesture they were performing was having some kind of effect, and that belief would in turn increase the effect that the programmed trigger has on the user.  And so a "virtuous circle" loop is created between user and application where one has an effect on the behavior of the other. 

Edit: the following information posted by Intel staffer Colleen regarding a feature in the R2 version of the SDK for estimating heart pulse rate using skin color changes may be of interest to those who want to investigate this subject.  Colleen said:

*****

"The R2 RealSense SDK contains this feature [pulse estimation] as part of the Face modules:  Pulse estimation tracks subtle change in face skin color over time and estimates the person's pulse rate. The algorithmic range shows as 30-60cm on the F200. And you can read samples at https://software.intel.com/sites/landingpage/realsense/camera-sdk/2014gold/documentation/html/index.html?manuals_facial_expression_data_2.html

[Vidyasagar_MSC]

Interesting!!! but how can we compare virtual reality and Augmented reality. Though they have few things in common, for me they are two different worlds.

[MartyG]

RealSense is regarded as being AR primarily because that's the way it has been marketed by Intel.  Once it is combined with a VR headset like Oculus Rift and the headset-wearer has a virtual representation of their hands or body (like with my project's full-body avatar) then it becomes VR.  That was what I primarily had in mind when I wrote the above article.

AR is great for many applications.  What it's not so great at is immersing you in a compelling fantasy environment.  It's hard to believe that you're in Westeros if all you can see in front of you is Wisconsin.  RealSense gives developers the chance to combine the best of AR with the best of VR to - like with Vive - explore environments where you can actually wrap your avatar fingers around virtual objects.  The difference there between RealSense and Vive though is that objects in a RealSense-powered simulation can be manipulated by closing your own flesh fingers, instead of clicking a button on a physical haptic-feedback hand-controller to close the digital hand.

[Karol_G_]

Yeah, I'm working on stuff like this. See this early proto:

https://www.youtube.com/watch?v=QtrPuYeh_NY

[Karol_G_]

Yeah, I'm working on stuff like this. See this early proto:

https://www.youtube.com/watch?v=QtrPuYeh_NY

[MartyG]

The fist fighting really made me laugh.  Thanks for sharing.  Great work!  :)