Community
cancel
Showing results for 
Search instead for 
Did you mean: 
Highlighted
Honored Contributor III
67 Views

Unity Tip: Simulating Flesh Sensitivity With RealSense and the Collision.Impulse Command

Hi everyone,

RealSense is ideally suited to interacting with content by using the real-world hands to manipulate that content with a virtual representation of the hands.  However, it is difficult to simulate in a digital environment the physical sensation of touch that is possible with the real flesh hands.

However, with a feature introduced in Unity 5.2 called Collision.Impulse, it is now very easy to translate data generated by the speed of collisions between a pair of objects into a descriptive status such as "Soft", "Painful" or "Very Painful".  In this guide, we will look at how to set up this tactile interface in Unity 5.2 and newer.

STEP ONE

The first step is to highlight in your Unity project the object that you want to give the ability to measure collision impact.  We will use the forearm flesh piece of our full-body avatar as an example.

1.jpg

STEP TWO

Create a new C# format script file inside the highlighted object, giving it a name that precisely describes the function of the script and the purpose of the object that it is placed within.  For the right forearm of our avatar, we chose the name 'CollisionForce_RightForearm'.

If you know that you are likely to be creating more than one instance of the same script (e.g for all the flesh pieces of an arm) then it is good practice to give all of the scripts the same word or words at the start of their name so that they will all be grouped together alphabetically in your scripts folder, making them easier to find without having to use a search-box keyword.

4.jpg

Ensure that you select the 'CSharp' option from the drop-down menu on the script naming window so that the script file you create is in the C# format necessary for the script that we will be writing, otherwise the script will not work if it is of another type such as 'JavaScript'.

5.jpg

STEP THREE

Once the script is created, go to the right-hand Inspector panel of Unity whilst your object is highlighted and find the section of the Inspector relating to the details of the script that you have just created.  

Right-click on the small gear-wheel icon beside the script name to bring up its menu, and select the 'Edit Script' option.

3.jpg

STEP FOUR

The script will open in Unity's script editor program, showing a default block of code that Unity places in every new C# script.

6.jpg

Delete everything in the script so that is completely empty and then paste in the C# script below.

using UnityEngine;
using System.Collections;

public class CollisionForce_RightForearm : MonoBehaviour {

public Vector3 Collision_Force_Right_Forearm;

public void OnCollisionEnter(Collision collision) 
{

Collision_Force_Right_Forearm = collision.impulse;

}
}

You will need to do some personal customizations to the script to make it suited to your project.

1. if you have given your script a name other than the 'CollisionForce_Right_Forearm' name in our example, you should edit the Public Class statement on line 3 to use the name of your own script in place of the word 'CollisionForce_RightForearm'.  For instance, if you called your script 'ImpactScript' then the Public Class line would look like this:

public class ImpactScript : MonoBehaviour {

2.  If you want to change the name of the variable that handles the impact data to be something other than the 'Collision_Force_Right_Forearm' variable name used in the example script, edit lines 4 and 7 to use your own variable name in its place.  For example, if you wanted to use the name ImpactVariable then the script would look like this:

using UnityEngine;
using System.Collections;

public class CollisionForce_RightForearm : MonoBehaviour {

public Vector3 ImpactVariable;

public void OnCollisionEnter(Collision collision) 
{

ImpactVariable = collision.impulse;

}
}

We will go through the script step by step to explain what each section of it does.

using UnityEngine;
using System.Collections;

These two opening lines, which are present in all C# scripts created in Unity. tell Unity where to locate the resources necessary for it to run the script.

public class CollisionForce_RightForearm : MonoBehaviour {

The 'Public Class' line of a script can be thought of as creating a container that the data that the script generates will be stored within, like a plastic tub (the container) that contains a certain type of food (the data).

public Vector3 ImpactVariable;

In a script, you can create a sub-container called a Variable that you can place inside the main container.  In these sub-containers, you can only place a certain type of data, just as you would not be likely to mix together different types of food in the same storage container in real life (you would only mix them once you were ready to combine them into something that can be eaten).

Common types of variable include:

float (stores decimal numbers)

int (stores decimal numbers)

bool (stores logic instructions such as True or False)

string (stores alphanumeric data such as "hello" or "hello12345").

In this script, we use a variable type called 'Vector3'. By doing so, we are telling Unity that we want the data to be stored not as a single word or number, but as a series of three numbers representing the three-dimensional directional axes X (horizontal), Y (vertical) and Z (depth).

public void OnCollisionEnter(Collision collision) 
{

If you have done Unity programming before, you may be familiar with the 'OnTriggerEnter' instruction, that activates a certain command when an object enters the boundaries of the "collider field" collision detector that surrounds the object that the OnTriggerEnter script is hosted within.

OnCollisionEnter is a type of 'On Enter' instruction that you may not have come across before.  Instead of activating an event when an object penetrates the collider boundaries of an object, the event is activated when the object colliding with the field simply touches the field - it does not have to penetrate inside that field.

Because OnCollisionEnter is preceded by the word 'void', this denotes that it is a Function.  A Function will only run the code that is contained within its opening and closing brackets - { and } - and ignore anything that is outside of the bracket except for variables like the 'Collision_Force_Right-Forearm' one that we defined in the header of the example script.

If you have done some basic experimentation with Unity scripting, you will know that scripts are made up of a header section, and then 'Function' sections of code that run differently depending on the type of Function used:  

Start() - runs the code contained within its { and } brackets only once and then stops.

Update() - runs the code  contained within its { and } brackets over and over in an infinite loop until something is done to stop it running.

Custom function names that you have defined yourself, such as Impact() - these are treated by Unity as though they are Start() type scripts and run only once before stopping.

Collision_Force_Right_Forearm = collision.impulse;

Finally, we tell Unity that the Vector3 type X-Y-Z axis data that should be stored inside the variable that we defined should be generated by the new 'collision.impulse' instruction that was introduced in Unity 5.2.

At the time of writing this guide, there is very little documentation for the instruction, other than this short explanation:

"The total impulse is obtained by summing up impulses applied at all contact points in this collision pair.  To work out the total force applied, you can divide the total impulse by the last frame's fixedDeltaTime".

A plain-English way of writing this is that when a pair of objects collide with each other, Unity generates an impact value for all of the axes of the collided-with object that were touched by the colliding object.  So although the collision.impulse instruction can detect collisions in the X, Y and Z directions, if the collision only takes place on the X axis then the impact values for the non-touched axes - Y and Z - will be zero.

STEP FIVE

In order for the script to respond to collisions, we need to add a 'Rigidbody' type physics component to the object that the script is hosted within.  A Rigidbody gives an object additional physics properties such as enhanced collision detection, weight drag and the ability to drop downwards under the influence of gravity.

Place a Rigidbody inside your object by highlighting it and then going to the 'Component' menu option and selecting the sub-options 'Physics' and 'Rigidbody'.

8.jpg

The Rigidbody will be added to the listing of details for the object in the Inspector panel.

9.jpg

STEP SIX

Edit the default settings of the Rigidbody with the settings shown below.

- Angular Drag to 0

- Use Gravity to unticked (disabled) so that the object does not fall downwards off the screen as soon as the project is run.

- Ticks placed in all of the Constraints boxes to prevent the object from moving or rotating at all when another object collides with the collider field of the object that the Rigidbody is hosted within.

STEP SEVEN

It is time to run the script for the first time.

Highlight the object that the script is hosted within to display the details of the script in the Inspector panel.  You will see that a new section has been added that contains the name of the Vector3 type variable that we defined, followed by three text boxes representing the X, Y and Z directional axes.

7.jpg

Run your project and use the controls (such as an arm whose movement is driven by the RealSense 'TrackingAction' SDK script) to collide the object containing the script and Rigidbody against another object.  This second object does not need to have a script or a Rigidbody inside it - it only needs to have a collider field surrounding it so that it can have an effect on our scripted object when the two collide with each other.

In our example, we swung the arm of our avatar against the side of the avatar's torso to generate a reaction.  The Collision.Impulse instruction in the script converts this impact into numeric values in the X-Y-Z text boxes in the script's Inspector panel settings.

10.jpg 

In general, a soft collision produces small values such as '0.05' and '0.5', whilst a faster (more powerful) collision produces higher values such as '2'.

STEP EIGHT

Having successfully created a means to measure collision intensity, we now need to convert the raw data into a format that can be more easily used to measure how soft or hard an impact is and trigger events based on those statuses (e.g playing a 'yelp with pain' audio clip if the data's numeric range goes into the 'Very Painful' zone when an avatar runs into a chair, or a small sigh of contentment if their virtual flesh is touched tenderly).

In our own project, we incorporated these conversion calculations into our project's "World Controller" that monitors the most important variables and uses logic statements such as If / And / Else to trigger events based on the current status of those variables/

https://software.intel.com/en-us/forums/realsense/topic/611270

Bearing in mind that not everyone will want to add this degree of complexity to their own project though, we will explain how to add the conversion system to the impact detection script.

Return to the header section of the script, where we defined this line:

public Vector3 Collision_Force_Right_Forearm;

Directly beneath this line, add the following line to define a new 'string' type variable, called 'Sensation_Lannel_Right_Forearm' in our example (as before, you can give the variable your own preferred name if you wish)..

BTW, the word 'Lannel' refers to the name of the avatar character who the arm belongs to.  We state the character's name in variables so that we can re-use the same script for multiple characters and know which variable relates to which character. 

public string Sensation_Lannel_Right_Forearm;

We added the word 'Sensation' to the front of the variable name to remind us that this variable contains data that is different from the one containing the raw X-Y-Z collision data.  As the name suggests, we want this variable to turn those numbers into a meaningful interpretation of them - the "Sensation" that the object that has been collided with is feeling.

The word 'Public' means that the value of the string will be displayed "publicly" in the script's settings in the Inspector panel.  If the term 'Private' were used then the variable would not be shown in the Inspector.

It is useful to make your most important variables Public and set the ones whose status you do not need to monitor as 'Private'.  This makes the data easier to follow at a glance in the Inspector panel, as the list of variables is not cluttered by ones whose values you have no interest in.

STEP NINE

You can optionally set the string to have a default value when the project is first run by defining an initial status for the variable.  This is done by adding a 'Start()' type function to your script, which - as the name suggests - will run once when the project is first run and then stop.

public void Start()
{

Sensation_Lannel_Right_Forearm = "Neutral";

}

In the above Start() block of code, we tell Unity that the initial contents of our Sensation variable should be the alphanumeric plain-English word 'Neutral'.  In other words, until the object actually touches something, it starts off having little feeling of sensation in it.

As mentioned above, this step is totally optional.  If you do not include it then the Sensation variable will be displayed in the Inspector as a blank box when the program is run, until the first time that a collision occurs.

STEP TEN 

Define an infinitely-looping 'Update()' type function in your script if you do not have one already.

public void Update()
{


}

In-between the opening and closing brackets, we are going to define a logical "If" statement that continuously checks the X-Y-Z axes of the Vector3-type variable and makes something happen if those values should fall within a certain range.

Insert the following If statements between the brackets:

public void Update()
{

if (Collision_Force_Right_Forearm.x >= 0.2 && Collision_Force_Right_Forearm.x <= 2) {

Sensation_Lannel_Right_Forearm = "Soft";

}

if (Collision_Force_Right_Forearm.x >= -2 && Collision_Force_Right_Forearm.x <= -0.2) {

Sensation_Lannel_Right_Forearrm = "Painful";

} 

if (Collision_Force_Right_Forearm.x >= -6 && Collision_Force_Right_Forearm.x <= -2) {

Sensation_Lannel_Right_Forearm = "Very Painful";

}


}

We arrived at the range of values to put in the brackets by trial and error.  We did repeated collision tests of different speeds whilst watching the X-Y-Z values in the Inspector and from this worked out appropriate ranges for the kind of values that were generated by a light, medium or heavy impact.  then we assigned to each of these numeric ranges a plain-English string word that described those values.

When the impact value falls beneath a certain range, the condition that the If statement is testing for is met, and it assigns the descriptive label to the Sensation variable that is defined directly beneath the particular successfully satisfied If statement.

Looking at the variable name that we have used in the If statements ('Collision_Force_Right_Forearm'), you may notice that there is an additional term written beside them - a full stop, followed by an 'x'. This tells the If statement that it should only be analyzing the values of a particular collision axis - the 'X' axis in this case - and ignoring the data in the other two axes.

The reason that we do this is to give the best chance of the If condition being met.  The more conditions that have to be satisfied, the harder it is to successfully trigger an event attached to the If statement.  So if the values all had to fall within specific ranges for the X, Y and Z axes before the 'Sensation' variable could be changed then it might not ever change.  By limiting the analysis to a single axis, we give the system the best chance of working correctly.

You can change the axis that is being checked by the If statement simply by changing the letter on the end of the variable after the full-stop - e.g

.y to check the Y axis collision data

.z to check the Z axis collision data.

Depending on the direction from which an object containing the impact script is typically approached in your own project, you may find that using one axis triggers the If statement more reliably than other axes.  Experiment to find the best one for your project.

If you run your project and watch the status of the 'Sensation' variable in the Inspector whilst you control your object (e.g by moving the hand if it uses an SDK script such as TrackingAction) then you should see its status change depending on how strongly you collide that object against another object.

12.jpg

You can add further If statements to your project that read the current status of the plain-English 'Sensation' variable and launch particular events, such as the audio files reflecting satisfaction or pain that we mentioned earlier.  For example:

If Sensation variable = "Soft" then play the "soft sigh" audio clip.

For the more painful sensations, you could alter the status of the movement variables in an avatar character to move slower and more awkwardly for a certain period of time, or play a "hobbling" walk animation.

Once you have successfully given one object impact sensitivity then you can use it as a template to add the system to additional objects.  At the time of writing, our World Controller global monitoring system can monitor sensitivity on the forearm, wrist and upper arm of both the avatar's left and right arms, with plans to add it to the other flesh areas such as the legs and face.

14.jpg

If you have any questions about this guide, please feel free to ask them in the comments below.  Good luck!

0 Kudos
0 Replies