Is there any documentation on what exactly you can do with the realsense data in unity?

[kkill3]

I'm trying to see if I buy a real sense camera if there is any in depth information on how to use the data from the camera. I cannot find a single youtube video, github document or anything that really explains what is capable with this beyond just using the data to pick up and interact with objects. I want to do things like move objects in unity realative to the person being tracked and other things like the data for depth tracking a person so i can use that data in unity to determin if a person is turned around, lifted their leg. Is there anything that explains how to use this data in unity?

[MartyG · Accepted solution]

The main documentation for the Unity compatibility 'wrapper' for RealSense SDK 2.0 can be found at the link below.

https://github.com/IntelRealSense/librealsense/tree/master/wrappers/unity

The RealSense SDK 2.0 software is compatible with the SR300, D415 and D435 camera models.

If you want to move objects, it is recommended that you use the camera with skeletal tracking software called 'Nuitrack SDK'. It costs $29.99 and also has a free trial version, and it has Unity integration. You can read about it in the discussion linked to below:

https://forums.intel.com/s/question/0D50P00004DbcqcSAB/i-want-to-implement-human-pose-estimation-from-the-video-data-coming-from-intel-realsense-depth-camera-d435-is-there-any-unity-plugin-available-for-this-kind-of-project-thanks-in-advance-cheers?language=en_US

[kkill3]

would you reccomend the tracking camera or depth camera if I need to single out the closest person to the camera if there are multiple people in the cameras view and also track there skeleton? can depth camera track multiple skeletons and use depth so that in unity i can grab data that represents the closest skeleton to the camera?

[MartyG]

You should get a depth camera such as the D415 or D435, as the T265 Tracking Camera does not have depth tracking.

Yes, the Nuitrack software can track ​more than one person. This is demonstrated in its Unity sample program for skeletal tracking on an RGB image.

https://download.3divi.com/Nuitrack/doc/UnityRGBandSkeletons_page.html

[kkill3]

Thank you, Probably the wrong place to ask but do you know if the nuitrack software has the ability to recognize partial skeleton and track that. say one or multiple people were not in the full FOV for entire skeleton tracking, lets say they come into the cameras view close with just their head. Can it track just peoples heads coming into the camera with just their head?

[MartyG]

The best place to ask is likely to be at the Nuitrack forum.

https://community.nuitrack.com/

Nuitrack at least has the ability to detect when parts of the body are hidden, or "occluded" and provide the user with feedback about the occurrence of that partial occlusion - an example being the head tracking being obscured by the hands.

A user called Westa who is expert on both Nuitrack and RealSense said in a Nuitrack forum post, "The way that the Nuitrack system works is by using algorithms (AI like) to recognise that there is a body in the field of view - and then uses further algorithms to calculate where parts of the skeleton for that body *likely* are".

"I use the term LIKELY ARE - since its all a set of approximations based on the knowledge of anatomy and human movements and inverse kinematics type math. For this all to work optimally - you need access to reliable and repeatable depth data. As the errors increase in the depth data - the accuracy of the reporting of the location of the points on the skeleton becomes less accurate from frame to frame which leads to jitter and jerky movements".

The full comment that the above quotes are extracted from can be found here:

https://community.nuitrack.com/t/experiment-setup-for-good-skeleton-tracking/666/8