Unity Tip: Improving Two-Handed RealSense Control Via Action Script Editing

[MartyG]

Hi everyone,

In previous guides in this forum, I have explained how to edit script files supplied with the RealSense SDK, such as TrackingAction, to improve performance and add new functionality.

In this guide, I will explain how I edited the setup of the SDK scripts to provide enhanced two-handed control.

If you have developed a RealSense project that tracks both the left and right hands at the same time, you may have found that the movement of objects assigned to Index 1 is more erratic and awkward t control compared to Index 0 objects.  This is because the scripts are continuously juggling which is the active hand (left or right) depending on which one is more visible to the camera.  

As the camera favors Index 0 as the primary hand, this means that Index 1 objects (the ones controlled with the "other hand" during two-handed control) tend to be more prone to having the tracking of that hand be lost, stalling the movement of Index 1 objects completely or making them pause and then resume movement.. 

However, if you could trick the camera into treating both the Index 0 and Index 1 hands as primary hands, then the smoothness of the controls would be the same for the objects assigned to both hands.  To do this, we can effectively assign control of Index 1 to a separate set of tracking scripts, identical to the original set except for a slight change in filename.

Important Note: make sure that you create a backup of your Unity project folder before using this guide, so that you can revert back to the previous version of the project easily if anything should go wrong.

STEP ONE

We will be creating copies in Unity of the following SDK scripts:

- TrackingAction

- HandTrackingRule

- HandDetectedRule

These three files are the primary scripts that govern the control of objects via tracking of the hand.  

Create a cube object in your project to act as a temporary storage container for the new script files that we are going to create.

STEP TWO

Create three C# script files inside the cube, using the above filenames with an extension on the end to distinguish them from the originals.  In my own project, I chose the extension _Index2.  So my three new script files inside the cube were:

- TrackingAction_Index2

- HandTrackingRule_Index2

- HandDetectedRule_Index2

STEP THREE

Open both the originals of the three scripts and your newly created scripts in the Unity script editor.

STEP FOUR

On each of the three original scripts, go to the 'Edit > Select all' menu option in the Unity script editor and then select the 'Copy' option to copy the entire contents of that script into its newly created '_Index2' counterpart script, replacing the block of default code that was placed in the script when it was created..

STEP FIVE

Select the Find and Replace option.  For the 'Find' text box, put in the script's original name (e.g HandDetectedRule').  In the 'Replace' text box, put the new name of the copy of that script (e.g HandDetectedRule_Index2).

 

The reason that we are doing this is to redirect the camera away from the original set of scripts and to the new separate set that will govern the movement of Index 1 objects during two-handed control.  By doing so, both hands will be treated as primary hands, because each hand communicates with its own dedicated script set.  It is as though the instructions for the left and right hand are in two separate universes.

STEP SIX

Once you have done the Find And Replace operation on all three of the new scripts, all that is left to do is to edit your new copy of the TrackingAction script at lines 117 and 120 to point the filenames to the copies of the two other files instead of the originals.  In other words, change 'HandDetectedRule' and 'HandTrackingRule' to whatever filename you used for your copies.  

STEP SEVEN

Locate the script copies in the 'Assets' section of your Unity Assets panel, where they will have been placed by default when they were created.  We are going to move them to the same folders that the original scripts are in.

Hold the left mouse button down on each of the new scripts in the Assets panel and drag them over to the appropriate folder in the 'Project' listing of folders.  The TrackingAction copy will go in the 'Actions' folder, whilst the copies of HandDetectedRule and HandTrackingRule will go in the 'Rules' folder.

STEP EIGHT

You can now delete the temporary cube that you created to store the new scripts in, as they are now safely filed away in the folder structure and the cube is no longer needed.

STEP NINE

Drag and drop your new copy of the TrackingAction script into the object that you want to be controlled by Index 1.  If you have a TrackingAction in there already, copy its settings over to the new TrackingAction and then remove the original from the object.

Remember to give the new TrackingAction an index number of '1' to indicate that it is an object that moves when both hands are visible to the camera. 

STEP TEN

It is time to do a test run.  Run your project and move both hands in front of the camera.  If all has gone well then you should notice a considerable improvement in the smoothness of the Index 1 controls when the Index 0 hand is being used at the same time.  For instance, before putting this system in place the am that is outstretched to the side (the Index 1 arm) could never reach sidewards anything like as far as the distance in the post-change image below.

Best of luck, and if you have any questions then please do ask them in the comments below!

[MartyG]

During further experimentation with my enhanced two-handed control setup, I decided to experiment with replacing the TrackingAction's 'Update()' type function with another type of infinite-looping function called OnGUI.  An advantage of OnGUI is that it checks the instructions within its code block several times per processing frame instead of the once per frame check that Update() does.

In theory, this change would make movement of TrackingAction-powered objects faster and smoother, just like a videogame that runs at a speed of 60 frames per second is more pleasant to watch than one that runs at 30 frames per second.  And indeed, when I did some test runs there did seem to be a movement speed increase of at least 50% without any jerkiness.

It should be pointed out again that using OnGUI() for this non-GUI purpose is not exactly "correct" programming.  The only price to pay for its use seems to be the generation of a very occasional non-harmful white information message in the Unity debug console (white messages seem to be information for the developer about something that is occurring in the background of the application rather than warnings or errors).

So if you want to achieve a movement speed boost without having to do a multiplication sum on the line of the TrackingAction that controls rotation, OnGUI may be a good option for you.

Note that Unity is very fussy about spelling - in order for it to work, you must write the function exactly as shown here - a capital O, and capital GUI.

Edit: whilst OnGUI is traditionally used for rendering visual elements such as option buttons, it could be argued that using it to control objects is a form of visual interface and so using OnGUI for that purpose would be "correct" programming after all, arguably.  It's just a different kind of user interface!

[MartyG]

Update: having proven the use of OnGUI to be stable for a single object (the rotation joint of my full-body avatar's hand-driven shoulders), I proceeded to make every other TrackingAction-powered part of the avatar (face expression, and waist and leg bending) use OnGUI too.

I was expecting a lot of white information messages to be generated in the Unity debug console, since logic dictated that the more components that used OnGUI, the greater the chance that the messages would be generated.  But in actual fact, I couldn't generate the message in the console at all no matter how much I tried, even though the number of TrackingActions in the avatar that were using the OnGUI function had increased significantly.  This would seem to back up my initial conclusion that it was completely safe to use OnGUI in place of Update with no ill effects.

The same speed increase that had been achieved in the shoulders was also now represented in every part of the avatar, with legs bending faster but just as smoothly as in the old system, and facial features (eye movement, eyebrows, lip movement, etc) were more responsive, more closely mirroring the human face's movement in real-time. This was because the OnGUI function, which runs the TrackingAction's checks of the human body positions several times a frame instead of just once per frame, could respond faster to the movement of landmarks instead of having to wait til the next frame before it could update the position or rotation of the TrackingAction-controlled objects.

With use of OnGUI, the avatar's camera-driven body, limb and face movements were now comparable to an animated movie or a high-speed videogame such as 'Street Fighter V'.

[Derp_K_]

Looks interesting

[MartyG]

Just wanted to provide a quick update for this article.  Although using the 'OnGUI' function in place of 'Update' in the TrackingAction script does provide a speed boost, there is a better alternative: FixedUpdate.

The advantage of using FixedUpdate in the TrackingAction is that it gives the same speed amplification that OnGUI does, but more stable performance.  This is because FixedUpdate is designed for use with physics-based Unity actions such as object rotation.  It processes the physics independently of the project's frame rate.  This means that you get very smooth movement, because not being tied to Unity's frame rate calculations means that movement lag is minimized.

Having said that, FixedUpdate should be used appropriately and not put in every single script.  Use FixedUpdate for physics and motion related code, and use the ordinary Update for rendering and drawing activities.