Hi everyone,
If you have been using the RealSense camera with Unity and SDK scripts such as TrackingAction, you will undoubtedly know that when an object has been moved or rotated with the hands, once the camera loses sight of a hand (the Hand Lost condition) then the camera-controlled object remains in whatever orientation it was in until the next time that the camera sees (the Hand Detected condition).
In many applications this will not be a problem, but in some it definitely is, especially with the control of avatar characters and their arms. This is because once hand tracking is lost, the arms will hang unrealistically in mid-air until tracking is regained. In the real world, the arms would drop downwards when control effort ceased.
We can though use a combination of custom-made scripting and the SDK's 'SendMessageAction 'script to automatically drop a limb section smoothly to its default position.
STEP ONE
Highlight the camera-powered object that you want to be able to reset upon hand tracking loss. In the case of my project, that is the lower arm joints of my avatar's arms.
STEP TWO
Create a new C# script in the object and give it a name appropriate to its purpose, such as 'ResetLowerArm'.
Once the script is created, go to the right-hand Inspector panel of unity whilst your object is highlighted and find the section of the Inspector relating to the details of the script that you have just created. Left-click on the tick-mark beside the script's name to remove it. This will make the script inactive, as we do not want it to be running until the Hand Lost condition has been met.
Next, right-click on the small gear-wheel icon beside the script name to bring up its menu, and select the 'Edit Script' option.
STEP THREE
The script will open in Unity's script editor program, showing a default block of code that Unity places in every new C# script.
Delete everything in the script so that is completely empty and then paste in the C# script below.
using UnityEngine;
using System.Collections;
public class ResetLowerArm : MonoBehaviour {
public void Update () {
GetComponent<ResetLowerArm>().enabled = true;
GetComponent<TrackingAction>().enabled = false;
transformer.rotation = Quaternion.Slerp(transform.rotation, Quaternion.Euler (transform.eulerAngles.x, transform.eulerAngles.y, (270 - transform.eulerAngles.z)), Time.deltaTime*2);
StartCoroutine (PauseTracking ());
}
IEnumerator PauseTracking() {
yield return new WaitForSeconds (2);
GetComponent<TrackingAction>().enabled = true;
GetComponent<ResetLowerArm>().enabled = false;
}
}
STEP FOUR
You will need to do some personal customizations to the script to make it suited to your project.
1. In this line:
public class ResetLowerArm : MonoBehaviour {
If you have called your script by a name other than 'ResetLowerArm', place that name in the line above in place of the word ResetLowerArm or the script will not run,
Also change the name in the two other occurrences of this script name in the script:
GetComponent<ResetLowerArm>().enabled = true;
GetComponent<ResetLowerArm>().enabled = false;
2. The script is configured by default to reset the rotation of the Z axis of the object. If you want the X or Y axes to be reset instead, replace this line:
transformer.rotation = Quaternion.Slerp(transform.rotation, Quaternion.Euler (transform.eulerAngles.x, transform.eulerAngles.y, (270 - transform.eulerAngles.z)), Time.deltaTime*2);
With:
X AXIS RESET
transform.rotation = Quaternion.Slerp(transform.rotation, Quaternion.Euler ((270 - transform.eulerAngles.x), transform.eulerAngles.y, transform.eulerAngles.z), Time.deltaTime*2);
Y AXIS RESET
transform.rotation = Quaternion.Slerp(transform.rotation, Quaternion.Euler (transform.eulerAngles.x, (270 - transform.eulerAngles.y), transform.eulerAngles.z), Time.deltaTime*2);
3. At the end of the rotation lines, you can increase or decrease the speed of the rotation by changing the multiplication ( * ) value beside the time.deltaTime statement.
If you increase the script's default value from '2' to a higher value, the reset rotation will move faster and reach its destination sooner.
If you remove the multiplication sum from the statement completely then the rotation will be slower.
Time.deltaTime);
4. The script is set by default to reset the angle of an object to 270 degrees. That happens to be the default angle for my avatar's elbow joint, but you can change it to whatever angle you want the object to return to from the position it is currently in when hand tracking is lost.
For example, to have it reset to a 90 degree angle, change:
transform.rotation = Quaternion.Slerp(transform.rotation, Quaternion.Euler (transform.eulerAngles.x, transform.eulerAngles.y, (270 - transform.eulerAngles.z)), Time.deltaTime*2);
To:
transform.rotation = Quaternion.Slerp(transform.rotation, Quaternion.Euler (transform.eulerAngles.x, transform.eulerAngles.y, (90 - transform.eulerAngles.z)), Time.deltaTime*2);
WHAT THE SCRIPT DOES
To aid in understanding the script, we will go through it step by step to explain what is happening.
1.
public void Update () {
GetComponent<ResetLowerArm>().enabled = true;
GetComponent<TrackingAction>().enabled = false;
Getcomponent<ResetLowerArm>.enabled = true activates the reset script (remember, we un-ticked its name to set it to be dormant by default).
Getcomponent<TrackingAction>.enabled = false deactivates the TrackingAction script in the object. This is necessary because if the TrackingAction is active, it will try to dominate the rotation of the object and prevent it from carrying out the reset instruction.
2.
transform.rotation = Quaternion.Slerp(transform.rotation, Quaternion.Euler (transform.eulerAngles.x, transform.eulerAngles.y, (270 - transform.eulerAngles.z)), Time.deltaTime*2);
This is our rotation reset instruction. Because it is listed in the script after the TrackingAction deactivation instruction, the script will not attempt to reset the rotation until after the TrackingAction has been deactivated, preventing the TrackingAction from disrupting the rotation.
What it is basically doing is saying to the object "If you deduct your current angle (eulerAngles.z) from the angle that I want you to go to (270) then you can work out how far you need your rotation to travel to get home".
3.
StartCoroutine (PauseTracking ());
}
IEnumerator PauseTracking() {
yield return new WaitForSeconds (2);
This section of code is called a Coroutine. It is like a script within a script. This coroutine uses a type of instruction called WaitForSeconds to instruct the script to pause its processing for 2 seconds and not do anything until that time period has elapsed. This is especially useful with infinitely looping Update() type functions like the one we are using in the reset script. Without this pause instruction, the script would try to activate every part of it almost instantly instead of doing it in a carefully timed order, resulting in a broken jumble of events.
The purpose of the WaitForSeconds instruction in this script is to give the reset rotation enough time to complete before the next set of instructions activates.
4.
GetComponent<TrackingAction>().enabled = true; GetComponent<ResetLowerArm>().enabled = false;
The final two instructions in the script, which execute only after the 2 second waiting period has elapsed, turn the TrackingAction back on so that hand tracking can resume immediately when the camera next detects the presence of the hand and then turns off the entire reset script, rendering it dormant again until the next time that the hand is lost.
STEP FIVE
As the reset script is inactive by default, we need a way to activate it only when the Hand Lost condition is met. We can do this with a script packaged with the RealSense SDK, found in the same folder as the TrackingAction script, called 'SendMessageAction'.
Drag and drop the SendMessageAction script into the object that is being reset (it must be in the same object as the reset script).
To begin configuring it to respond to the hand, left-click on the 'Add Triger' (sic) menu to drop it down and select the 'Event Source' option fromt he menu.
A sub-option called 'Event Source' will be added to the SendMessageAction's interface. Left-click on the 'Add' button to drop down a menu of options for different ways to trigger an event, and select the 'Hand Lost' option.
This will instruct the SendMessageAction to activate our reset script (by Sending a Message to it) only when the hand is no longer visible to the camera.
STEP SIX
The following step is optional depending on whether you are using a single hand or two-handed control with your project. If you are using two-handed control, we recommend making this change in order to get better response from the controls when both hands (Index 0 and Index 1) are visible to the camera at the same time.
Left-click on the arrow beside the name of the 'Hand Lost' option that has now been added to the SendMessageAction to expand open its hidden options. Left-click on the 'Access Order Near To Far' menu to drop down a list of alternatives and select 'Access Order Fixed'.
If you are only using one-handed control in your project, it is fine to leave this option on the default 'Access Order Near To Far'.
STEP SEVEN
Finally, go to the 'Function Name' setting at the base of the TrackingAction and change the default setting of 'OnTrigger' to the name of the function within the reset script that you want to send an activation signal to.
This can be somewhat confusing for new RealSense developers, as they expect to be sending an activation to a script name, rather than the name of a function *within* a script file.
In our reset script, our main rotation function is within an 'Update()' type function, so type the word 'Update' into the text box. Do not include the two brackets - ( ) - that normally come after a function's name.
STEP EIGHT
We are now ready to test out the reset function. Run your project and move your camera-controlled object away from its default start position.
In the case of my project's avatar, that means raising the lower arm upwards.
Once the hand is moved out of the view of the camera and the 'Hand Lost' condition is triggered, the SendMessageAction script recognizes the hand loss and sends an activation instruction to the Update() function within the reset script. The reset script then tells the object to deactivate its TrackingAction, rotate back to the default start position and then reactivate the TrackingAction so that camera control of the object can resume.
If you have any questions about this guide, please feel free to ask them in the comments below. Good luck!
Link Copied
