Need hand tracking explanation

[Raditya_U_]

Hello guys

This is a snippet code from RealSense VStudio Tutorial, project 3_Hands_Tracking the gesture recognition section

		int numOfGestures = outputData->QueryFiredGesturesNumber();
		if(numOfGestures>0){
			// iterate through fired gestures
			for(int i = 0; i < numOfGestures; i++){
				// initialize data
				PXCHandData::GestureData gestureData;
				// get fired gesture data
				if(outputData->QueryFiredGestureData(i,gestureData) == PXC_STATUS_NO_ERROR){
					// get hand data related to fired gesture
					PXCHandData::IHand* handData;
					if(outputData->QueryHandDataById(gestureData.handId,handData) == PXC_STATUS_NO_ERROR){
						// save gesture only if you know that its right/left hand
						if(handData->QueryBodySide() != PXCHandData::BodySideType::BODY_SIDE_UNKNOWN)
							renderer->NotifyGestures(handData->QueryBodySide(),gestureData.name);
					}

				}
			}
		}

Is the QueryFiredGestureData() that recognize the gesture we are making? I check the definition and I found

virtual pxcStatus PXCAPI QueryFiredGestureData(pxcI32 index, GestureData & gestureData) const = 0;

How that function check the gesture we are making? I mean, I guess the program check the coordinates of our hand from an existing data to find out what gesture we are making, then where is the data? I want to make a new gesture. My plan is adding my own coordinates to the existing data, along with other gesture to make a new one. It is allowed by Intel, right? I just don't know where to find it. Any ideas?

-R

[Dagan_E_Intel]

Hi Raditya,

If I understand your question correctly, you are asking where is the information of the hand joints? the 3d positions, so that you can detect your own gesture?

Lookup the function QueryTrackedJoint in the manual, and see how it is used (you can look at the code in the samples).

in the example you quoted it will look something like this, for getting the 3d coordinates of the index-finger-tip:

if(handData->QueryTrackedJoint(PXCHandData::JointType::JOINT_INDEX_TIP) == PXC_STATUS_NO_ERROR)
                            {
                                std::printf("X: %f, Y: %f, Z: %f \n",jointData.positionWorld.x,jointData.positionWorld.y,jointData.positionWorld.z);
                            }

[Raditya_U_]

Wow that is really helpful, Dagan!

Now I can determine my own gesture by comparing the thumb, index, and middle finger world coordinates

Thank you!! :D

[Scott_W_1]

I'm missing something here - I inserted the code you gave above into the tutorial example and I keep getting error messages:

Error    4    error C2660: 'PXCHandData::IHand::QueryTrackedJoint' : function does not take 1 arguments   

Here's my snippet:

for(int i = 0 ; i < numOfHands; i++)
        {
            // get hand joints by time of appearence
            PXCHandData::IHand* handData;

            if(outputData->QueryHandData(PXCHandData::ACCESS_ORDER_BY_TIME,i,handData) == PXC_STATUS_NO_ERROR)

            {
                PXCHandData::JointData jointData;
                // iterate through Joints and get joint data 
                if (handData->QueryTrackedJoint(PXCHandData::JointType::JOINT_INDEX_TIP) == PXC_STATUS_NO_ERROR)
                {
                    std::printf("X: %f, Y: %f, Z: %f \n", jointData.positionWorld.x, jointData.positionWorld.y, jointData.positionWorld.z);
                }
                for(int j = 0; j < PXCHandData::NUMBER_OF_JOINTS ; j++)
                {
                    handData->QueryTrackedJoint((PXCHandData::JointType)j,jointData);
                    nodes = jointData;

                }

            }
        }

[Jack_K_1]

Scott,

The documentation says QueryTrackedJoint  has a second argument JointData.

Syntax

C++
 pxcStatus  QueryTrackedJoint(JointType label, JointData &data);
 
Parameters

 
 label
 The joint label. See the JointType enumerator for definitions.
 
 
 data
 The joint data, in the JointData structure, to be returned.
 

Regards,

Jack

[zhenxi_c_]

HI SCOTT,do u know how to recognize some new gesture except the default 10 gestures（V，fist，spread finger etc）？ it have read the program for about two weeks and still do not know how to modify the parameters or using the raw data to recognize the sign language gestures.........

waiting for your reply..........

[MartyG]

RealSense does support the creation of new camera algorithms by developers to expand its capabilities beyond those supported by default in the RealSense SDK.  However, it has been said that it is not as easy to do so as it was in RealSense's direct predecessor, 2013's Perceptual Computing SDK and the accompanying Creative Senz3D camera.  This added difficulty was apparently as a result of the increased complexity of RealSense.

Backing this up is an Intel blog article by Code Monkeys developer Chris Skaggs, where he says the following:

"A more “user-friendly” system comes at the cost of granular control.  Developers have a lot less access to raw data in the Intel RealSense SDK and customizing processing algorithms is no longer a simple matter.  In the end, though, the Intel RealSense SDK is a major improvement over Intel Perceptual Computing at basically every level.  And while the nerdcore coder in us miss the unfettered data stream [in the PerC SDK], the deadline-oriented coder is grateful for the improved level of accessibility and productivity."

https://software.intel.com/en-us/articles/to-realsense-w-unity

So it should be possible for you to create new camera recognition algorithms to make sign language possible.  It will take a bit of work to learn how to do so though!

Edit: an alternative to developing a new camera algorithm would be to create a virtual body in the Unity game engine with RealSense and place invisible trigger points around its body.  When a certain combination of factors were triggered simultaneously, that could be recognized as a sign language gesture.  

For example, if the programmed logic detects that the arm is crossing a certain point in front of the chest whilst the index and middle fingers are pressed together (detected by collision fields on the sides of the fingers) then the program could say "That must be 'X' sign language gesture) and initiate a certain response to that gesture being recognized.

Here's a 3 month old video of my company's RealSense-powered full body avatar in action to help illustrate the concept (the avatar is even more capable now in the present day).

https://www.youtube.com/watch?v=bGGlbc0yk2E