Aiming the sensor downwards?

[PKlok]

While testing we found out the sensor works a lot better when it's aimed straight forward on a 1-1-5meter height. However in our case, we would like to aim it downwards. We have big screens which we want to place it on. On the top of the screen aiming downwards

Something like this:

But then aiming it angled downwards. Is that possible at all? We noticed a lot of loss in leg tracking, during a skeleton tracking minigame/demo. So for now we have it under the screen pointing straight forward but that is not possible when launching this product.

We use the Realsense D415 with Nuitrack behind it by the way.

Greetz,

Patrick

[MartyG]

RealSense cameras can be pointed downwards. The results can vary depending on the application though. For depth sensing it usually works fine, and Intel even offered a 3D printing blueprint for creating a downward-facing RealSense mount for their Ready To Fly drone kit.

https://www.instructables.com/id/Intel-Aero-Drone-Downward-Facing-RealSense-Camera/?language=en_US

For joint tracking, a downward-facing orientation can introduce problems, depending on how that joint tracking was implemented. For example, in the previous generation of RealSense cameras (F200 and SR300 and the '2016 R2' SDK), the design of the joint tracking module meant that the joints could be tracked from above in the same way that they were when the hand was straight-ahead. But you first had to show the palm to the camera in straight-ahead pose to activate tracking, as the hand module depended on recognizing a rough shape resembling a palm space before the camera would follow the joints.

The strategy of mounting the camera atop a TV facing downwards has been used successfully in the past for a project called 'Tanked', where a RealSense R200 stereo camera (a predecessor of the D415 stereo camera) was attached to a TV and faced down into a box of sand, rearranging a virtual representation of the sand as the real sand was moved by the player hands.

https://www.youtube.com/watch?v=tz0Aa4qjxeQ&language=en_US

I faced the same issue as you with leg tracking when trying to develop real-time joint tracking with the '2016 R2' SDK for a PC game. I never implemented a solution, though I did consider having the user wear a belt with two RealSense cameras attached, one facing upward for hand joint capture and one facing downward for leg joint capture. On the old 2016 SDK, its joint detection algorithm searched for areas that looked like joints rather than something that looked like the actual human bones, so it treated knees and foot soles as palms and the toes as fingers!

I recall I have an old test video on my YouTube channel where I highlighted this with my RL foot:

https://www.youtube.com/watch?v=IOJtApQqa8A

[PKlok]

Thanks for the reply. So this leans towards a no for us? Having the user wear a belt is a big-no and the camera is aimed at an angle down. So not straight down like Tanks(which looks awesome though!)

This results in the camera recognizing users and having a decent upperbody tracking, but horrible legg tracking.

If anyone knows anything it's highly appreciated.

[MartyG]

In late 2018, Intel published a seminar on body tracking with the 400 Series cameras that may yield better results for you than skeletal tracking with Nuitrack in this case.

https://www.youtube.com/watch?v=VSHDyUXSNqY

[PKlok]

That's interesting! I watched the whole video and that looks really solid. Sadly no solution for us as we only have a simple realsense camera to work with and introducing anything new hardware wise is not an option.

[MartyG]

If you are trying to detect the entire body (upper and lower) with a single camera view, I wonder if you would get better results from using a D435 camera instead of a D415, as the D435 has a wider field of view (85.2° x 58° x 94, compared to 69.4° x 42.5° x 77° on D415) and so can fit more of the body into its view.

[PKlok]

I too questioned this. However, the user seems to be fully in view. It's purely bad tracking.

[MartyG]

Nuitrack also recommend avoiding dark clothing. So if you are scanning a person wearing black business trousers, that could be a factor in your bad leg scanning.

https://download.3divi.com/Nuitrack/doc/Preparations_page.html#preparations_subsection7

[PKlok]

Yeah, we already tried. Somewhere in a post on their forum one of the Nuitrack employees recommended someone to aim the camera straight forward like we have atm. So even they know it doesn't work a 100% when aiming it angled downwards

[MartyG]

Would it be an option to position it under the TV set and use a shallower downward angle closer to straight-ahead to compensate for the reduced height? Even straight-ahead might capture the whole body at that height.

On the old RealSense R200 camera model, it had a magnet on the back so that you could attach it to a monitor's casing without having to use a mounting bracket. It came with a plate that you could put over the magnet when the camera wasn't attached to the screen. Though I guess a magnetic mount-point may count as 'extra' hardware that your project does not permit.

[PKlok]

It's certainly an option, The problem is, there's already stuff ordered/told to customers.

We had tradeshows a while back and mounted them to the bottom of the screens with a little 3D printed mount. Worked fine. But the eventual idea, something that has already been communicated by the company is that it will be on top of the screen. Especially because it's both for business calls like Skype etc, and our skeleton project which i'm making. So while it's an option we're looking into any possibilities to keep it mounted on top of the screens.

[MartyG]

If the camera has to ideally be elevated on top of the TV and angled downwards, and Nuitrack acknowledge there is an issue with detection when sensing at an angle, AND you do not want to break assurances to customers, then it is a difficult issue to solve.

In this situation, the most achievable solution may be to use Inverse Kinematics (IK). This is where the positions and rotations of some joints can be predicted based on the angles and positions of joints they are attached to that can be tracked. In the case of Nuitrack, you could track the higher-up left and right hip joints and predict the orientation of the lower knee joint based on them without actually tracking them. So if you are not trying to track the knees, you could use a shallower angle for the camera atop the TV.

In my own project I use a similar IK-style system that I custom-built to generate very realistic full-body virtual avatar motion in the Unity engine without tracking every body point.

Here's another old test video where an avatar moves its upper and lower legs based purely on the animation of the hip joints. As a rough formula, the lower legs bend to 50% of the travel distance that the upper legs are bending, since anatomically a large upper leg lift is accompanied by a smaller lower-leg swingback.

https://www.youtube.com/watch?v=HHzXdLqI8p4