RealSense Object Collision Technique In Unity - Final Prototype

[MartyG]

Hi everyone,

You may have read in another forum post that my company was developing a technique for Unity objects moved with camera inputs to be able to physically collide with each other instead of passing through intangibly.  We have now finished our prototype and will share our findings in this post.  First though, here is a video of the system in action.

https://www.youtube.com/watch?v=5Ihw4RSYAWk&feature=youtu.be

The basic principle of the system is that it uses a sphere that is fixed in position but is free to rotate in any direction on the spot like a ball-bearing when another object makes contact with its collider.  This spinning motion generates collision physics that gently repel the object that collided with it.

It is similar to how the gravity on planets works.  The rotation of the planet's high mass generates a gravitational field.  The larger and heavier the planet and the faster it rotates, the stronger the gravitational force generated is.  It is thought that in future, space stations will be able to generate artificial gravity by rotating.  The sci fi series Babylon 5 used giant rotating rings to produce gravity for its station's inner environments.

We have written a step by step construction guide below for those who want to try to use it in their own Unity project.  If not, we hope that the video above inspired you! 

*****

STEP ONE

Create a 'Sphere' type object in Unity and place it at the point on your rotating / moving camera-controlled object where you want collision detection to take place.  

As you can see in the above video, we placed multiple spheres in such a way that the collider fields that they projected covered most of the surface of our avatar.

Size the sphere so that its collider field extends about 3 millimeters above the surface of your camera-driven object, like a force-field.  Ensure the sphere is placed in such a way that the collider projected by the sphere comes out of your camera object on all sides, otherwise the object may have collision detection on some sides of it but not others.

An alternative approach to the sizing is to use a small sphere with a very large collider field that is large enough to penetrate the skin of the object it is buried within and provide our all-surrounding 3-millimeter forcefield..

To use a Ghostbusters reference, it is okay to "cross the streams" and have the colliders of each sphere in a set overlap each other to provide total surface coverage.  But we found that this reduces the effectiveness of the collision detection.  In our avatar's torso, we used two large spheres for the upper and lower halves of the body and left a small gap in their colliders between them so they did not overlap.  Doing so made the collision detection extremely reliable.

When the sphere-equipped object (e.g a point on an avatar arm) deflects, the object will try to move through the object being collided with at any point where there is a gap in the sphere coverage.  So for example, if you had just one collision sphere on the upper arm of an avatar and one on the lower arm, the opposite arm would be able to pass through the un-covered area inbetween the two spheres' collider coverage.  

Another important principle is that for two camera-controlled objects to collide with each other and stop instead of passing though, each object must have a collision sphere equipped.  For example, if only one of our avatar arms had the spheres installed then when the arms made contact with each other, they would pass through each other despite the spheres on one arm.  The same is true for the hands.

Edit: I thought I should add the rationale for why a sphere is used.  It is because no matter what direction the sphere rotates in when it is struck by a colliding object, its spherical collider field will always remain in the same position as it rotates with the sphere.  But if you use a different shape of object such as a rectangle, with a box-shaped collider, then the position of the collider will change (e.g if a rectangle object with a box collider tips 45 degrees then the rectangular collider will tip 45 degrees with it like a see-saw, opening up a gap in the collision coverage for objects to pass through.  But with spherical collider fields, the object being protected by the collider is always covered no matter which way the sphere rotates when hit.  

STEP TWO

Child-link the sphere to the nearest camera-driven object to it.  In our avatar in the video, that is the shoulder and elbow rotation joints on the arms, with the spheres on the lower arm childed to the elbow joint and the upper arm's spheres to the shoulder joint.

http://test.sambiglyon.org/sites/default/files/bearings.jpg

This principle is *very* important.  The reason for this is that we want the collision spheres to rotate with the camera-driven object and not get left behind when the object moves.  So if you had a set of spheres on the lower arm of the avatar, you would child then to the elbow joint so that when the lower arm moves, the spheres move with the arm as though they are glued to them and remain in place, because the same camera script that is moving the arm is simultaneously moving the spheres and keeping sphere and arm in perfect positional sync.

When creating collision spheres for finger sections, child the sphere to the nearest camera-driven joint.  So in a finger with three rotation joints in it, the sphere nearest the base would be childed to the bottom joint.  The sphere in the middle of the finger would be childed to the middle joint.  And the sphere at the end of the fingertip would be childed to the fingertip rotation joint.  See the video for an illustration of this principle.

STEP THREE

Give every single sphere a 'Rigidbody' physics component with the following settings.

*****

Mass: 10000

Drag: 1

Angular Drag: 0

Use Gravity: enabled

Is Kinematic: disabled

Freeze Position: X, Y and Z ticked

Freeze Rotation: all boxes unticked

*****

A fast way to do this is to make one sphere and then copy and paste it and move it to a different position and ensure that it is childed to the correct camera object.  the 'Rigidbody' component and its settings will be copied with it.

STEP FOUR

Disable the collider field of every object that is within the borders of a sphere collider.  This is because the colliders can interfere with the sphere's colliders and render collision detection unreliable or non-existent.  The spheres will provide the collider for the objects instead.  In Star Trek terms, it is like a starship extending the borders of its energy shields around a second ship positioned beside it to protect it because its own shield has failed.

STEP FIVE

Create a blank transparent PNG texture in your art software and import it into your textures folder in Unity, then drag and drop it onto each of your spheres and give them a 'Transparent' lighting type to make the spheres invisible so that, like in our video, the collision system cannot be seen.

An easy way to make a transparent PNG in your art software is to start a new blank image and do ';Select All' and 'Cut' to cut out the white background.  Then save your image in PNG format.

FINALLY

Run your Unity program and play around with the collision, adjusting the positions and sizes of the spheres and their colliders until you have got the collision detection reliability as good as you can get it.

Best of luck!

[Vidyasagar_MSC]

That's awesome...Thanks for sharing Marty

[MartyG]

Thanks again Vidyasagar for your encouragement.  ^_^

Further experimentation today found that another application for the collision spheres is restricting the movement of an object in a certain direction to provide greater control over moving objects with the camera.

For instance, if you put one sphere over the top of or beside the camera-driven object then you can change the behavior of the object and force it in a different direction than it would normally go in.  By trying the sphere out in different positions around the object, you can make it steer in different directions.

A second application is to prevent extreme mis-rotations, such as an avatar shoulder popping out of the body during a big arm swing.  You can do this by attaching one sphere to the mis-rotating object, with the sphere's collider protruding from its surface, and put a second sphere in the vicinity of the direction that the object tries to move in when mis-rotation occurs.

http://test.sambiglyon.org/sites/default/files/blocker2.jpg

Once the "blocker" is in place then when mis-rotation occurs, the camera-driven object will bump up against the collision sphere and stop.  Using this system, I was able to greatly reduce the occurrences of shoulder pop-out in my avatar during big arm moves.

Edit: in a subsequent test, it was found that if you use a sphere-collider with a tall rectangular Box type collider and put it behind your camera-driven object then you can minimize occurrences of the object leaping backwards when the hand or face is first detected - if the object tries to do so then it bumps up against the box collider and stops.

http://test.sambiglyon.org/sites/default/files/blocker3.jpg

[MartyG]

Still continuing my experiments into physical collision of RealSense-driven objects.  My latest finding is that giving each of the collider spheres' Rigidbody physics component a high Drag value of 100 (instead of the '1' we were using before) makes the avatar limbs and hands much more smoothly-moving and controllable.  

This is because the Drag value makes the colliders "sticky" and stops an object from bouncing strongly away from it when there is a collision.  This helps a lot when trying to do precision positioning of an avatar's arms and hands.

[MartyG]

Here's my latest findings on getting optimum collision results from camera-controlled objects such as an avatar in Unity.

*  The solidity and reliability of collisions seems to be greatly enhanced if the object has a Rigidbody with the following settings:

Interpolate: Extrapolate

Collision Detection: Continuous Dynamic

There is a trade-off for this reliability though, as these settings can slow down the movement of an object a little.  If you use it for one or two objects, the slowdown is not too serious.  If you use these settings in multiple objects though (e.g all the flesh pieces down an avatar arm from shoulder to wrist) then the slowdown multiplies.

A good way to get around this is to be smart with the objects you give these settings to.  For example, if an avatar arm is made up of three pieces merged within each other to sculpt a particular arm shape, put the above settings only in the widest of these pieces, as - if the other pieces are submerged within the arm - the widest piece will be the piece that collides with something most often, and so giving the collision settings to the submerged pieces would be a waste as they would rarely be collided with by other objects, because the wide piece always reaches the surface of the other object first.

*  It is a good idea to use as few pieces as possible when sculpting an object.  Developers who are new to object modeling tend to try to make the most complex construction they possibly can.  But often, you can achieve the same appearance with less pieces.

My company's squirrel-guy avatar has used three to four flesh pieces for the upper and lower arms and wrists for a while now.  We thought that this complexity was terribly important, even though it actually introduced far, far more to go wrong once the avatar started moving and Unity's physics interactions kicked in.

Looking at how the arms were constructed today though by highlighting submerged pieces, we could see that most of them were barely visible on the surface of the arms and 99% of their surface was hidden within the widest part of the arm.  So we stripped the arms and wrists down to a simple one piece assembly containing only the widest part of each.  When we did a comparison of before and after, we found that the visual difference was barely perceptible.

http://test.sambiglyon.org/sites/default/files/reduce.jpg

When we did a test run of the camera-controlled limbs though, there was a definite improvement in speed and accuracy as the unnecessary baggage was shed!

*  When we gave the Extrapolate and Continuous Dynamic settings to the Rigidbodies in all of the rotation joints and flesh pieces in our avatar fingers, the gain in stability of finger bending was remarkable too.

*  We do not recommend using these settings in facial-tracking objects though.  These are difficult enough to move using the user's facial movements as it is (as the face motions are much smaller than those produced by the user's hands), and giving them the Extrapolate and Continuous Dynamic settings makes them even more difficult to move.  They are definitely best used with hand-tracking objects.

[MartyG]

My latest work on sphere-based collision systems for camera-controlled objects (see the posts above) indicates that placing collider spheres so that they overlap with each other enhances collision detection considerably.  This is contrary to earlier findings that suggested that overlapping them weakened collision detection.

Our theory for why this may be the case is based around how the spheres are placed in relation to each other.  In our earlier tests, two large we had two large spheres - one overlaid on the chest and one overlaid on the stomach - and the lower stomach sphere overlapped the base of the upper one, forming an '8' shape where their collider fields curved inwards at the middle point whee one field ended and the other began. 

More recent research suggests though that the collision between the collider sphere and another object (such as an avatar hand) registers more strongly when the collision takes place against a relatively flat area (such as the side of a large sphere object) rather than against a curve like that on a small sphere, where there is only a small area for the colliding object to make contact with because the collider field is curving away from the impact point directly above and below it.

We found though that if we positioned several small spheres so that half of the field overlapped halfway across the field of the sphere next to it, it formed a collision field surface that resembled the bumpy tread of a vehicle tyre / tire.  

@@@@@

Objects that collided against this pattern of field seemed to be stopped significantly more regularly.  The likely reason for this was that when an object such as an avatar hand struck the field, its length spanned across a few overlapping fields, producing an effect akin to touching one long object, because the object was unable to slip between gaps in the collision coverage (like the '8' shape we had between the chest and stomach colliders) and pas through the body/

When we employed the above technique on the body torso, using three large overlapping collider spheres instead of two in order to reduce the size of the area of the field that curved inwards to a minimum, occurrences of pass-through in that part of the body were reduced even further. 

We also added additional overlapping spheres to our avatar's fingers, and the number of times in which the hand caught on whatever they were pushing against instead of passing through intangibly also increased.

[MartyG]

My latest research on collision physics in Unity involved developing a policy where the collision spheres attached to my avatar's body were divided into two categories of Rigidbody settings in order to minimize physics lag during the avatar's movement.

Category 1: Interpolate = ''Extrapolate' and Collision Detection = 'Continuous Dynamic'

These settings were given to the Rigidbody of collision spheres attached to avatar objects that (a) frequently experienced high-speed collisions, and (b) had a possibility of becoming trapped behind the back of the avatar and spoiling the user's play experience.

'Extrapolate' smooths out collisions by predicting what the next frame of movement will be, dampening them so that they are gently repulsed from the surface of another object instead of bouncing off uncontrollably (which makes it a lot more difficult to accurately position the arms.)  It is especially appropriate for use in Rigidbody-equipped objects that have jerky movement.

'Continuous Dynamic' helps to stop an object colliding with another object at high speed from passing through that object.  The 'None' collision setting lacks this high-speed protection, and so 'Continuous Dynamic' provides an additional amount of protection against pass-through occurring.  The price of this protection is that it generates more physics lag than 'None' does and so should be used sparingly, hence the reason for developing the setting selection policy in this post.

Our research also indicated that only the object being collided with needed the 'Extrapoliate' smoothing, not the object doing the collding.  So this meant that the collider spheres attached to body parts such as the chest, midriff, stomach, groin and legs were placed in Category 1 to receive the full settings.  

The collider spheres of the arms, meanwhile (the objects doing the colliding with the torso and upper legs) were placed in Category 2 with the lesser physics settings.  

The lower legs were placed into the lesser Category 2, since it was hard for the avatar hands to reach down that far and so the lower legs were unlikely to need the collision smoothing offered by the 'Extrapolate' setting.

Although the arm spheres received the lesser setting, the palm and finger spheres were given the full Category 1 settings.  This was because the hands and fingers can collide with each other at high speed when the player brings the hands together, so they needed the Cat 1 collision settings to ensure that the hands did not wildly bounce off each other when they were brought together by the player.

So in summary, these are the parts covered by Category 1:

Chest

Midriff

Stomach

Groin

Lower legs

Fingers

Category 2: Interpolate = ''None' and Collision Detection = 'Continuous Dynamic'

Collision spheres placed in Category 2 only received the 'Continuous Dynamic' collision setting, with the 'Interpolate' option set to 'None.'  The avatar parts whose collision spheres received the lesser protection include:

Upper and lower arms

Lower legs

Conclusion

When we thoroughly tested our avatar with these new categorized settings, collision resistance was stronger and the limbs moved faster as physics lag was reduced.