Unity tip: Controlling Avatar Arms And Legs With A Single RealSense Camera

[MartyG]

Hi everyone,

During development of my company's RealSense-powered full body avatar, one of the priority goals has been to have both the arms and legs able to be controlled by a single camera.  

The major obstacle to doing so though is that if your avatar has both left and right arm control (as ours does), both of the hand indexes are used up.  So if you set your left and right arm to track the '0' and '1' indexes, using those same indexes for the rotational joints of your avatar's legs means that when the arms are moved, the legs move too, and vice versa.  What one is really aiming for is independent control of the arms and legs.  Sadly though, until dedicated indexes for the legs and toes are incorporated into the SDK, this is a mightily difficult goal to achieve.

We have at least succeeded today though in using the 'TrackingAction' script's Rules system to set up a mechanism where control can be switched at will between the arms and the legs, freezing the arm movement whilst allowing the legs to be moved and then returning control to the arms.  Here's how we did it.

STEP ONE

Set your arm joints to have their Start rules as 'Hand Opened' and 'Hand Detected'.  Set the Stop rule as 'Hand Closed.'

STEP TWO

Set your leg joints to have their Start rules as 'Hand Closed' and their Stop rule as 'Hand Opened.'

HOW THE RULE LOGIC WORKS

(a)  When the program first begins, it detects the real-life hands as they are held up to the camera and activates the 'Hand Detected' rule to grant initial control to the avatar arms.

(b)  When the user closes their hands into a fist, the 'Hand Closed' Stop rule is initiated and the arm joints freeze.

(c)  At the same time, the leg joints see the hand-closed gesture and activate the 'Hand Closed' rule in their Start rule, enabling the leg joints to start moving.

(d)  When the user opens their real-life hands, the 'Hands Opened' rule in the leg joints' Stop settings is activated and tracking of the hands ceases, freezing leg joint movement.  

(e) At the same time, the 'Hands Opened' rule in the Start settings of the arm joints see that the real-life hands have been opened and activate movement, returning control to the arms whilst the leg joints are frozen until the next time that the closed-hand gesture is detected.

The basic reason why the arm joints have the 'Hand Detected' rule in their Start settings is that when the program is first run, the hands need the 'Hand Opened' rule to start moving.  BUT it is often the case that the tracking does not wake up until you push the palms of the hand towards the camera when the program begins.  

Without that action, it is very hard to get the camera to detect the 'Hands Opened' rule.  So having the 'Hands Detected' rule in the Start conditions ensures that as soon as the real-life hands are opened, the camera immediately sees them and wakes up the tracking of the arm joints.

CONCLUSION

Switching control between the arms and the legs is not the ideal situation for us.  We want both the arms and legs to be able to be controlled simultaneously yet also completely independently.  We are optimistic that we will find a solution eventually.  But until then, the above solution probably represents the best way to incorporate dual arm and leg control into an avatar without the two conflicting their inputs and trying to move each other!

[MartyG]

We found that switching back and forth at will between arm control and leg control by closing and opening the real-life hands was actually quite fast and intuitive, with the delay no greater than moving the fingers from a joypad 'Punch' button to a 'kick' button.  Our aim is still to achieve real-time control of both arms and legs simultaneously, but this compromise system works well until then.

Forgive the avatar sky-floating in the video below - we haven't programmed the body to drop down after it jumps into the air yet.  :)

https://www.youtube.com/watch?v=CT56zP3FP1U&feature=youtu.be

[MartyG]

Here's a video of version 2 of our avatar arm-leg control system, where we found that we could achieve a much more stable level of control over the leg simply by using a basic two-joint arrangement where the first constraint was constrained to 'X' rotation and the second was constrained to 'Z'.

https://www.youtube.com/watch?v=4nUkuqzGDJ0&feature=youtu.be

Production of V.2 reminded us of an important lesson.  Although the leg could swing in any direction, the amount of conscious effort required to steer it meant that it wasn't actually much *fun*.  This is a common trap in development - making something that is incredibly advanced yet does not contribute much to the ease of use or enjoyment of the application.

With our 2-joint X-Z arrangement, we achieved a similar range of movement to the uber-complex V.1 but made it simple enough that we could move the opposite arm at the same time without thinking too hard.

That leads us to the second important development lesson.  In V.1, both legs could be controlled at the same time.  This worked great if only one leg was being used at a time, but if both legs were swung up in the air at the same time then the avatar just floated in mid-air until the legs were moved down to the floor again.  This completely broke the user immersion.  

We spent a day working on a prototype of a gravitational system in Unity (the complex nature of the avatar's multiple parts and joints meant that just turning on gravity in the avatar's top-most parent object just made the avatar fall apart).  It worked but was far too awkward to activate, so we went back to the drawing board to re-think.

We asked ourselves how often the end-user of our avatar would actually want to put both legs up in the air simultaneously.  Unless they were doing martial arts or WWE moves, it seemed like most people would only want to move one leg at a time.  Even in fighting games, only one leg is usually used for a kick move whilst the other leg remains on the ground.

We therefore decided to have both arms controllable simultaneously as that was vital to our gameplay, but only have one leg be movable with the camera.  In doing so, we would ensure that the game was fun for the user by giving them the power to swing a leg if they felt like it (it isn't necessary to do well in the game) without them having to stop thinking about the game world in order to concentrate totally on pulling off the move.

If a game's feature isn't fun to use, it isn't worth it!

Edit: we know that it's good to provide a choice of legs to use in the same way that either the left or right arm can be used on our avatar as the primary arm.  If we can work out a system for keeping one leg on the floor whilst the other leg is in motion then we'll definitely support that (we did have a working prototype of this but it made the arm control unreliable, so we'll keep working on the problem!)