Hi everyone,
If you have developed a project with the RealSense camera in Unity, you may have noticed that when using face tracking, turning your real-world head left or right can cause a face-tracked object to move when you did not want it to. This is a variation on a similar issue with hand tracking, where turning the wrist left-right can mis-rotate an object even if that object is not set up to follow the wrist joints of the real-world arm.
In my own project, involving the control with the camera of a full-body avatar that can crouch down to the ground when the head is lowered and raised, this was a particular problem. Turning the head left and right would cause the avatar's waist to bend up and down. As a user typically makes many unconscious small movements of the head, this made precise positioning of the avatar's hands problematic, as well as making the controls seem somewhat unwieldy.
I have now developed an approach that addresses the head mis-rotation issue well, and I will share it in the guide below.
STEP ONE
The first requirement is an object containing the RealSense SDK's TrackingAction hand and face-tracking script that is set up to rotate left and right like a person's head when the real-world head is turned left and right. I already had such a joint in the head of my avatar, but creating one from a simple sphere object works just as well.
Find an empty location in your project where you will be able to create your sphere without it being visible to the main camera and then go to the 'GameObject' menu at the top of the Unity window and select the sub-options '3D Object' and 'Sphere' to create a sphere object at your chosen location.
STEP TWO
We now need to add the TrackingAction script component to the sphere. Go to the Assets > RSUnityToolkit > Actions' folder of the Unity 'Assets' panel, hold the left mouse button down on the 'TrackingAction' script in that folder, and drag it over to the sphere in the central panel of Unity and release the left button to drop the script into the sphere.
If you do not have the RSUnityToolkit folder in your Unity project, you can add it by going to the location on your computer where the RealSense SDK is installed and opening the following folder:
'Intel > RSSDK > framework > Unity'
Open your project in Unity and then double left-click on the file named 'UnityToolkit'. This will automatically install the RSUnityToolkit folder into your Unity project.
STEP THREE
Going back to our sphere ... once the TrackingAction script has been dragged and dropped into it, highlighting the object will display the TrackingAction script's default settings in the right-hand 'Inspector' panel of Unity.
Left-click on the 'Hand Tracking' menu next to the 'Set Defaults To' label to drop the menu down, and select the 'Face Tracking' option.
The settings on the TrackingAction will update to display the configuration for face tracking instead of hand-tracking.
Left-click on the 'Constraints' option to expand it open, revealing a set of tickable boxes. Placing ticks in the boxes on the Position or Rotation rows prevents the object that the TrackingAction is hosted in - our sphere, in this case - from position-traveling or rotating in the ticked axis direction.
We want to configure the constraints so that the sphere is only able to rotate left or right when the real-world head is turned left or right, and it is unable to move from its position at all or rotate up / down and forward / back.
Logically, we would select the horizontal 'X' axis to turn the sphere left-right. One of the most important lessons that a developer who is new to Unity can learn though is that its axes do not always follow logic! You may find that the axis to turn left-right is actually Y or Z rather than the expected X.
A tip for finding the correct rotational axis quickly is to go to the 'Transform' section at the top of the Inspector panel, input a test value into the X, Y or Z box of the 'Rotation' settings and then press the Enter key and observe which way the sphere rotates. If it travels in the wrong direction, change the rotation value that you just inputted back to '0' and put the test value in another axis box until the sphere turns the correct way. You will then know the axis that should be left unlocked in the Constraint settings.
In my own project, the axis that produced left-right rotation was 'Y'. So in the Constraint settings, every box but 'Y' rotation had a tick placed inside them to lock them, leaving only the 'Y' axis free to move.
STEP FOUR
Having set up the TrackingAction to turn the sphere left and right when the real-world head is turned left-right, we are now ready to test whether it actually works.
You could create a game camera, point it at the sphere and then run your project in order to observe the sphere physically rotating as your head turns. This is a bit fiddly though, as you have to get the new camera correctly oriented, switch off the main camera so that it does not conflict with the new one you just made, and then disable the new camera and turn the main one back on again when your test is finished.
A much easier way to test the sphere for correct rotation is to highlight it so that its settings are displayed in the Inspector panel, then run the project and observe the live-updating Rotation values in the Transform section at the top of the panel as you turn your head left and right. If only the axis that you left unconstrained is changing in value and the other axes remain fixed at a value of '0', you know that the TrackingAction has been configured correctly.
STEP FIVE
Now that setup of the sphere's TrackingAction is complete, we want to give Unity a way to easily communicate with the sphere using scripting so that it can read the rotational angles of the object in real-time. A common way to do this is with an instruction called 'GameObject.Find'. This is not the best way to do it though. GameObject.Find runs slowly because Unity has to check an index for the object every single time that its processing "frame" updates which could be a couple of times each second if the script is using an infinitely-looping Update() type function.
The preferable method is to give the object a "tag name" and then look up the tag using an instruction called 'GameObject.FindWithTag'. This instruction is faster because it does not check Unity's index for the object each frame.
To tag an object, highlight it in the Inspector and go to the 'Tag' option at the very top of the panel. Beside it is a drop-down menu that is set at 'Untagged' by default.
Left-click on the 'Untagged' option to expand the menu open. You will see a list of pre-defind tags. Ignore all of these. Look for an option named 'Add Tag' at the very bottom of the list and select it.
Upon selecting the option, the list of tags will be displayed again, but now with the addition of '+' and '-' buttons. Left-click on the '+' to create a new slot in the list where you can type in a name to tag the sphere with. You can choose any name you wish - for example, Head Turn Detector'.
STEP SIX
With the tag defined, you can now add it to the sphere so that Unity can easily communicate with it. Highlight the sphere and again left-click on the 'Untagged' option to drop the tag list down. you will find that your newly defined tag has been added to the base of the list. Select it to set the sphere to have that tag name.
STEP SEVEN
With the sphere successfully created, tested for horizontal rotation and tagged, we can now begin constructing the scripting that will read the rotational angle of the sphere in real-time and freeze rotation if the real-world head is turned to the left or right of the front-center orientation.
Highlight the sphere and create a new C# format script file inside it. This can be done by highlighting the sphere and then going to the 'Component' menu of Unity and selecting the 'Add' option to open a pop-up window of options. You should select the 'New Script' option from the bottom of that list, select 'CSharp' as the script type from the drop-down menu and provide a name for your new script.
For my own project's script, I chose the name 'HeadRotator_LeftRightTurn_Freeze'. It is a good idea to give your script a name that describes what it does so that you can quickly find it again by searching for keywords such as 'freeze'.
STEP EIGHT
Right-click on the small gear-wheel icon at the end of the script's name to drop a menu down. Select the 'Edit Script' option from the menu to open your new script file in the Unity script editor, where you will see a default block of code that Unity places in every new C# script.
You can also open a script in the editor by double left-clicking on its icon in the 'Assets' panel if you wish.
Please note that the 'Public Class' line of the script contains a reference to whatever name you gave your script file. Because I named mine 'HeadRotator_LeftRightTurn_Freeze', that is the name used in this line. If you called your script another name, then it would be that name that would be shown in the default block of code. You must ensure that whatever name you gave the script exactly matches the name in this line or your script will not run. So if you called your script 'Herbert' then using the 'public class HeadRotator_LeftRightTurn_Freeze' line in your own script would cause it to fail, because Unity would not be able to find the script when it is actually named Herbert!
In order to ensure that our script can communicate with other scripts, and those scripts can communicate back, it is good practice to insert the world 'public' in front of the function names so that Unity does not regard your script as being private and closed-off. So 'void Update()' becomes 'public void Update()' and 'void Start()' becomes 'public void Start()'
If you have previously written JavaScript scripts in Unity, the Void instruction is the C# equivalent of Function. So JavaScript's 'function Start()' and 'function Update()' become 'void Start()' and 'void Update()' in a C# format script. Apart from the name change, they work the same way as Functions in Javascript.
There is a good reason why we are writing our script in the C# language and not JavaScript. Unity prefers that scripts of the same type communicate with each other, although it is possible to get a C# and JavaScript to communicate with some hard work. The RealSense SDK's TrackingAction script is written in C#, and so creating our script to talk to the TrackingAction in the C# language is the best way to guarantee a successful outcome.
STEP NINE
We are going to be adding code to all three sections of the default code block - the header, the Start() block and the Update() block. To make room for this code, use the Enter key to open up empty gaps in each section.
STEP TEN
We are now ready to tell the script how to locate the sphere that we created earlier by finding it via its tag name. We do this with a type of instruction called 'public GameObject'. The term Game Object tells Unity that it should be looking for an object and not a numerical value, alphabetic name, etc. We will place the instruction in the header section of the script above Start{} and refer to the sphere by the reference name 'detect_headturn'.
using UnityEngine;
using System.Collections;
public class HeadRotator_LeftRightTurn_Freeze : MonoBehaviour {
public GameObject detect_headturn;
// Use this for initialization
public void Start () {
}
// Update is called once per frame
public void Update () {
}
}
STEP ELEVEN
With this variable set that refers to the sphere, we must tell Unity that when it sees the variable, it should use a GameObject.FindWithTag instruction that contains the sphere's tag name to locate the sphere.
In the Start() section of code, we therefore insert the following instruction, which must contain the earlier-defined tag name inside quotation marks.
using UnityEngine;
using System.Collections;
public class HeadRotator_LeftRightTurn_Freeze : MonoBehaviour {
public GameObject detect_headturn;
// Use this for initialization
public void Start () {
detect_headturn = GameObject.FindWithTag ("Head Turn Detector");
}
// Update is called once per frame
public void Update () {
}
}
STEP TWELVE
Things get trickier here. We need to define in our script a "logic statement" so that the script can work out from the sphere's rotational angle whether it has turned to the left or to the right of the front-center point.
A logic statement is a set of conditions that use terms such as IF, AND, OR and ELSE to describe what should happen when a particular set of conditions that the script is programmed to analyze are met. For example:
IF the sphere is detected in the angle range that denotes that it has moved to the left side of its center-point, do something. ELSE (if the condition has not been met), do something else.
Unity can use a script to read the angle of any object in real-time by checking its rotation (also known as a Transform) by referring to that angle as a "eulerAngle" - a type of rotation that has four axes instead of the usual three X, Y, Z and W. So to check the angle of a particular axis, you can use the instruction 'transform eulerAngles', with a letter on the end denoting which axis should be looked at:
eulerAngles.x
eulerAngles.y
eulerangles.z
To maximize the precision of the check, we want to define a range of two angle numbers that the rotation of the sphere should fall between before the script tries to freeze the rotation of the sphere. When the angle is outside of that number range, the script should do nothing. If we only provided the script with a single angle to check (e.g 140 degrees) and told the script that it should freeze the sphere if the angle was detected to be 140 degrees or less, it would regard any angle between 0 and 140 as meeting the freeze condition, and the sphere would be frozen as soon as the project started running instead of only freezing when the real-world head was turned.
Because we want Unity to be constantly checking the angle, we need to place our If statement within the infinitely-looping Update() function of the script.
using UnityEngine;
using System.Collections;
public class HeadRotator_LeftRightTurn_Freeze : MonoBehaviour {
public GameObject detect_headturn;
// Use this for initialization
public void Start () {
detect_headturn = GameObject.FindWithTag ("Head Turn Detector");
}
// Update is called once per frame
public void Update () {
if (transform.eulerAngles.y > 140 && transform.eulerAngles.y < 170 || transform.eulerAngles.y > 180 && transform.eulerAngles.y < 210) {
}
}
What we are saying in the above script is that if the angle is between 140 and 170 degrees, or between 180 and 210 degrees, then the script should take action to freeze rotation (though we have not scripted that freezing mechanism yet).
The symbols && and || denote the logic instructions AND and OR respectively. So the above script line is saying, do something IF the angle is greater than 140 AND less than 170, OR the angle is greater than 180 AND less than 210.
STEP THIRTEEN
With our logic statement set up, we need to edit the TrackingAction script to be able to lock and unlock the open rotational constraint on the sphere when we send a freeze or un-freeze instruction to it depending on the current detected angle of the sphere.
Go to the Action sub-directory of the RSUnityToolkit folder where you drag-and-dropped the TrackingAction script from earlier. Double left-click on the script's icon in the Assets panel of Unity to open the TrackingAction script in the script editor.
Scroll down to line 200 of the script, where you will find the beginning of its Update() function. Click on the line directly above Update() and use the Enter key to create some empty space above it.
Paste the following block of code into the empty space'
public void FreezeMotion()
{
Constraints.Rotation.Y = true;
}
public void FreezeMotion_Cancel()
{
Constraints.Rotation.Y = false;
}
What this code is saying is that when our script containing the If logic statement sends an activation signal to the code block named FreezeMotion(), the 'Constraints.Rotation.Y = true' instruction should tick the Y rotation axis of the sphere's contraints, preventing it from moving.
When the logic script sends an activation signal to the FreezeMotion_Cancel() code block, the 'Constraints.Rotation.Y = false' instruction unconstrains the Y rotation axis, allowing rotation to resume.
Scripts that use non-standard user defined function names instead of the usual Start(), Update(), etc are treated as instructions that do not run when the program is run, and can only be activated by making a direct call to them with scripting. They also only run once and then stop until the next time that an activation signal is sent to them.
If you want to freeze / unfreeze an axis other than Y, simply change Y to X or Z. Ensure that you write it as a capital letter instead of lower-case, otherwise you will get an error.
STEP FOURTEEN
Now that the freeze-unfreeze routine is set up in the TrackingAction, we can return to our If statement in our angle checker and tell it how to communicate with the functions that we added to the TrackingAction.
Directly beneath the If statement in the Update() block, we place the following line of code:
detect_headturn.GetComponent<TrackingAction> ().FreezeMotion ();
using UnityEngine;
using System.Collections;
public class HeadRotator_LeftRightTurn_Freeze : MonoBehaviour {
public GameObject detect_headturn;
// Use this for initialization
public void Start () {
detect_headturn = GameObject.FindWithTag ("Head Turn Detector");
}
// Update is called once per frame
public void Update () {
if (transform.eulerAngles.y > 140 && transform.eulerAngles.y < 170 || transform.eulerAngles.y > 180 && transform.eulerAngles.y < 210) {
detect_headturn.GetComponent<TrackingAction> ().FreezeMotion ();
}
}
The format of the line that we added can be broken down as follows:
<TAG REFERENCE NAME>.GetComponent<SCRIPT TO TALK TO> ().<FUNCTION IN THAT SCRIPT TO TALK TO>();
So we are telling the script to:
(a) find the sphere via its tag name using the 'detect_headturn' reference name.
(b) Find the script called TrackingAction within that object.
(c) Activate the function named FreezeMotion inside the TrackingAction script.
STEP FIFTEEN
As well as telling the If statement what to do if the sphere's rotation angle falls between the numeric range that represents it being to the left or right of the front-center point, we also need to tell it what to do when the sphere is no longer in the zones that activate the freeze (i.e when the sphere is in the center-front orientation). We do this with an ELSE logic statement, which we insert below the IF statement.
The instruction placed within the Else statement tells Unity to send an activation signal to the FreezeMotion_Cancel routine inside the TrackingAction to set the Y rotation constraint to unlocked if the sphere's angle is not within the range of values that cause rotation to be frozen.
using UnityEngine;
using System.Collections;
public class HeadRotator_LeftRightTurn_Freeze : MonoBehaviour {
public GameObject detect_headturn;
// Use this for initialization
public void Start () {
detect_headturn = GameObject.FindWithTag ("Head Turn Detector");
}
// Update is called once per frame
public void Update () {
if (transform.eulerAngles.y > 140 && transform.eulerAngles.y < 170 || transform.eulerAngles.y > 180 && transform.eulerAngles.y < 210) {
detect_headturn.GetComponent<TrackingAction> ().FreezeMotion ();
}
else
{
detect_headturn.GetComponent<TrackingAction> ().FreezeMotion_Cancel ();
}
}
}
STEP SIXTEEN
It is likely that depending on how your Unity project is set up, the angles of the to-the-left and to-the-right positions may be different to the values listed in the scripts in this guide. We therefore need a way to easily test the range of angles that will activate the freeze routine only when the head is turned left or right.
To do this, we can add a "debug" test instruction to the script temporarily that will print a text message in Unity's Console panel when the If statement's conditions have been met. The debug instruction is placed directly beneath the If statement so that it only prints the message in the console when the condition is met.
using UnityEngine;
using System.Collections;
public class HeadRotator_LeftRightTurn_Freeze : MonoBehaviour {
public GameObject detect_headturn;
// Use this for initialization
public void Start () {
detect_headturn = GameObject.FindWithTag ("Head Turn Detector");
}
// Update is called once per frame
public void Update () {
if (transform.eulerAngles.y > 140 && transform.eulerAngles.y < 170 || transform.eulerAngles.y > 180 && transform.eulerAngles.y < 210) {
detect_headturn.GetComponent<TrackingAction> ().FreezeMotion ();
Debug.Log ("Frozen!");
}
else
{
detect_headturn.GetComponent<TrackingAction> ().FreezeMotion_Cancel ();
}
}
}
The line Debug.Log ("Frozen!"); tells Unity that if the sphere's rotation falls within the defined angle range, the word 'Frozen!' should be printed in the console panel.
STEP SEVENTEEN
Run the project, and turn your head left and right. As you do so, observe when the 'Frozen!' message is printed in the Console panel and where the position of your head is at the time that the message is triggered. By a process of trial-and-error adjustment of the angle values in the If statement, you should eventually arrive at a set of angles that only sends the activation signal to the 'FreezeMotion' routine in the TrackingAction when your head is clearly in the to-the-left or to-the-right position.
The trigger point should also be far enough to the left or right of center-front that the user is not constantly mis-triggering the freeze. It should though also not be so far to the side that the freeze action is rendered pointless because the objects in the project that are being mis-rotated by head turning have already carried out that rotation by the time that the freeze activates.
There is no need to define a separate set of angles for the center-front position. The ELSE statement tells Unity that if the freeze condition is not met, then it should constantly be sending an activation signal to the 'FreezeMotion_Cancel' routine to keep the sphere's rotational axis unlocked.
Best of luck, and please feel free to ask questions in the comments below!
Link Copied
