How Swipe Work in New Camera?

[CLi37]

I just received the new camera. 

I tested the Hand View sample. However when I set gesture to swipe and test it there are only errors: alert out of boarder.

No any swipe. The thumb down and up is fine.

I inserted the camera before the SDK 3.0 installation and finished the setup fine.

Shall I need to update the firmware of the camera? I guess it is already the latest version.

I need to know what kind of gestures will be mapped to the touch or mouse operations.

i.e.

How to simulate the mouse operation like press left button down and drag move around or touch tap and move around. 
That is how to point an object and move it anywhere (mouse drag and move)?

How to pitch and zoom in two fingers?

How to swipe left and right?

Can gesture camera be a pointer device like mouse?

Suppose I want to point to an 3D object by my finger and rotate the 3D object by move around my finger, how can i do this? 

I can accurately swipe left and right by using first generation camera and sdk. 

But new camera and sdk might have been completely changed to a new swipe algorithm so far I can not work on swipe anyway.

[Xusheng_L_Intel]

Here is the instruction of swipe.

Swipe is triggered  when the user swipes to the left with right hand or swipes to the right with the left hand. User needs to be inside the FOV and after calibration so it's best to design the interactions that require swipe in a way that the user hand was already in the FOV before the swipe – that will insure you have calibration and would be more natural for the user. The swipe is moving the hand from one side to the other when the palm is facing the side and fingers are more or less towards the camera.

Please let me know if you still have issue to detect swipe.

Thanks!

[CLi37]

Followed your instruction to put hand inside FOV but still can not swipe. I can see the bright point in the hand but still no swipe in hand view sample. Could you please show me a video how to do swipe?

[germ13]

@chang-li just wanted to share, this document might help: https://software.intel.com/sites/default/files/managed/0f/b0/IntelRealSense-WindowsSDKGold_PB_1114-FINAL.pdf

It catalogs some of the supported gestures.

[Colleen_C_Intel]

Unfortunately swipe is a pretty common word with different interpretations. For the pre-built swipe gesture that's in the touchless controller section needs the full hand in FOV first - for calibration (say a high five) and then point the fingers towards the camera (but not like thumbs up) with the palm facing to the (middle in) side and move the fingers to the middle (LH move right, RH move left). 
 

[steve-vink]

I have found some of the gestures to be very difficult, especially in the limited field of view. Swipe is one of the difficult ones, and the pinch action I also find difficult to register. I don't think the camera is your problem. Make sure you are in a well lit room and keep experimenting until the gestures start to register. As mentioned earlier, it is always good to stay with a "high 5" to register the full hand.

[CLi37]

Once upon a time I can make the program to recognize swipe for a while.
Later I can not repeat it again. I have made sure my hand is inside the
boarders and calibrated.

A gesture need to be reliable and accurate for real applications.

tap, thumb up, v sign are reliable and accurate. But swipe and pitch are not.

The time serial gestures caused troubles that may be in algorithm. 

I guess the algorithm of swipe between first version SDK and current version SDK are different.

In the first generation SDK and Camera I can reliably work on swipe left/right by one hand.

[CLi37]

I have adapted code clips from hand-viewer sample to my code following the instruction article "Hand Analysis Tutorial".

I can capture the totalNumberOfGestures as 10. I used config->EnableAllGestures() and config->EnabledAllAlerts(). The problem is

outputData->QueryFiredGesturesNumber() is always returned 0 even if I made gestures in front of camera. 

I have done the same debug on hand-viewer sample it returned the right number of fired gestures.

I checked the CPU usage and I found that when I made gestures in front of camera the CPU usage was increased. That means the camera actually actived in the recognization of gesture.

What is wrong to return the zero number of gesture calling?

Do you have sample to run the code in "Hand Analysis Tutorial" without GUI code mixed?

I can not understand the following code clips:

https://software.intel.com/sites/default/files/managed/15/33/intel-realsense-hand-and-finger-tracking-tutorial.pdf

// stream data
while (psm->AcquireFrame(true)>=PXC_STATUS_NO_ERROR)
{
  outputData->Update();

  // create data structs
  PXCHandData::GestureData gestureData;
  pxcCHAR gestures[NUM_HANDS][PXCHandData::MAX_NAME_SIZE] = {};
  PXCHandData::BodySideType handSide[NUM_HANDS] = {PXCHandData::BODY_SIDE_UNKNOWN};
 
  // iterate through fired gestures
  for(unsigned int i = 0; i < outputData->QueryFiredGesturesNumber(); i++)
  {
    // initialize data
    wmemset(gestures, 0, sizeof(gestures));
    handSide = PXCHandData::BODY_SIDE_UNKNOWN;
    // get fired gesture data
    if(outputData->QueryFiredGestureData(i,gestureData) == PXC_STATUS_NO_ERROR)
    {
      // get hand data related to fired gesture
      PXCHandData::IHand* handData;
      if(outputData->QueryHandDataById(gestureData.handId,handData) == PXC_STATUS_NO_ERROR)
      {
        // save gesture only if you know that its right/left hand
        if(!handData->QueryBodySide() == PXCHandData::BODY_SIDE_UNKNOWN)
        {
           wmemcpy_s (gestures,PXCHandData::MAX_NAME_SIZE*2, gestureData.name, sizeof(gestureData.name));
           handSide = handData->QueryBodySide();
        }
      }
    }
  }
}

The outputData->QueryFiredGesturesNumber() is supposed to be 0 or 1?

Then how can I detected various gestures such as fist, thumb_up, .etc?

[Xusheng_L_Intel]

 The QueryFiredGesturesNumber function returns the number of fired gestures. You can try this NO GUI code and let me know if it works for you or not. Thanks!

class MyHandler:public PXCHandConfiguration::GestureHandler {
public:
   virtual void PXCAPI OnFiredGesture(const PXCHandData::GestureData &data) {
	   wprintf_s(L"getsure state is %d\n", data.state);  
	   wprintf_s(L"gesturedata is %s", data.name);
	   wprintf_s(L"*****");
	}
};

int wmain(int argc, WCHAR* argv[]) {
	// error checking Status
	pxcStatus sts;

	// create the PXCSenseManager
	PXCSenseManager *psm=0;
	psm = PXCSenseManager::CreateInstance();

	// enable hand analysis in the multimodal pipeline
	sts = psm->EnableHand();

	// retrieve hand results if ready
	PXCHandModule* handAnalyzer = psm->QueryHand();
	PXCHandData* outputData = handAnalyzer->CreateOutput();
	PXCHandConfiguration* config = handAnalyzer->CreateActiveConfiguration();
	config->EnableAllAlerts();
	config->EnableAllGestures();
	MyHandler handler;
	config->SubscribeGesture(&handler);
	config->ApplyChanges();
	config->Update();
	psm->Init();
	
	PXCHandData::GestureData gestureData;

	while (psm->AcquireFrame(false)>=PXC_STATUS_NO_ERROR) {
		outputData->Update();
		
		if(outputData->QueryFiredGesturesNumber()>0){
			memset(&gestureData,0,sizeof(gestureData));
			if(outputData->QueryFiredGestureData(0,gestureData)==PXC_STATUS_NO_ERROR);{
				// get hand data related to fired gesture
				PXCHandData::IHand* handData;
				if(outputData->QueryHandDataById(gestureData.handId,handData) == PXC_STATUS_NO_ERROR)
				{
					// save gesture only if you know that its right/left hand
					if(!handData->QueryBodySide() == PXCHandData::BODY_SIDE_UNKNOWN)
                      wprintf_s(L"Hand is %d\n", handData->QueryBodySide() );
				}
			}
		}
		
    // release or unlock the current frame to go fetch the next frame
		psm->ReleaseFrame();
	}
	// close the last opened stream and release any session and processing module instances
	psm->Release();
	return 0;
}

[CLi37]

Your sample code worked. But I still can not capture swipe once. Thus I changed to apply thumb_up/thumb_down as a replaced pair. But this is no natural. 

A problem of gesture control in RS is that there is no begin and end of a gesture and about the next gesture.

Another problem is that when a gesture action finished the second same gesture will not generate any reaction. i.e. the first thumb_up may have effect but do thumb_up again no reaction at all. Sometime one thumb_up may generate two or three actions. 

I also need a zoom/shrink pair of gestures (two fingers) to control what gestures of RS can do this?

And tap and drag/move gesture returned a coordinate.

So far my experience of using gestures is that they are unreliable and inaccurate. How can I deal with this? Any solutions? 

[Xusheng_L_Intel]

Please do not enable all the gestures and ONLY enable the gestures you need. You can change config->EnableAllGestures() to config->EnableGesture((L"swipe").

[CLi37]

After enable selected gestures "thumb_up/_down", the response speed is faster, but still not as fast as keyboard. A special DSP may be needed to improve performance.

Now thumb_up is a gesture then what is the next thumb_up? What is the show gesture -> release gesture -> show gesture again procedure?

Is spreadfinger and fist a pair of operation for zoom and shrink a view of a map?

[HexLord]

hi,

Could you tell me the role of    "outputData->Update();" in the program above?

Thanks

Shaleen

[Raditya_U_]

virtual pxcStatus PXCAPI Update() = 0;

Update() function return pxcStatus, and the definition is

/**
   This enumeration defines various return codes that SDK interfaces
   use.  Negative values indicate errors, a zero value indicates success,
   and positive values indicate warnings.
 */

all in pxcstatus.h

[Vijay_Aditya_B_]

How to simulate the mouse operation like press left button down and drag move around or touch tap and move around. 
That is how to point an object and move it anywhere (mouse drag and move)?

-->Can gesture camera be a pointer device like mouse?

yes u can do (simulating mouse is already there) it with "touchless controller list view sample code . its very easy also

-->But new camera and sdk might have been completely changed to a new swipe algorithm so far I can not work on swipe anyway.

yes there is no proper swipe gesture functionality yet there in 2014 sdk. i also need it 

[CLi37]

@Vijay Aditya B

I have read the "touchless controller list view" before, I found it was suitable for keyboard not mouse.  Any URLs/Code Clips to show the use?