Multi face detection

[NZafr]

Hi,

I am working with RealSense SR300, DCM 3.3.27.5718, and RSSDK 10.0.26.0396.

When my application detects more than one face, I can use RSSDK to tell me that more than one face is detected (QueryNumberOfDetectedFaces), but when I apply QueryLandmarks(), I only get landmarks information for the first face that was detected (the face that have index 0).

For all the other faces, QueryLandmarks() returns NULL.

Anyone encountered this issue?

Is it possible to get landmarks data for the faces that were not detected first? If yes, how?

Thanks.

[NZafr]

Thank you, Sergio.

Looking forward to hear from you.

[idata]

Hi noamz,

Thank you for your patience.See this link that shows the maxTrackedFaces property of the LandMarksConfiguration class: https://software.intel.com/sites/landingpage/realsense/camera-sdk/v1.1/documentation/html/index.html?landmarksconfiguration_pxcfaceconfiguration.html https://software.intel.com/sites/landingpage/realsense/camera-sdk/v1.1/documentation/html/index.html?landmarksconfiguration_pxcfaceconfiguration.html . Setting this property to more than "1" should enable the application to get landmarks for more than one face simultaneously. Unfortunately, the Face sample in SDK 2016 R2 does not use this property. However, the Face sample in SDK 2016 R3 (see file RSSDK/sample/face/Facetracking.cs line 155) does use it. If setting LandMarksConfiguration.maxTrackedFaces in SDK 2016 R2 does not solve the problem, then we'd recommend you to switch to the SDK 2016 R3 Face module. Note that in SDK 2016 R3 the Face sample Visual Studio solution file does not work. We'd need you to open the .cs file directly.Let us know if this answers your question.Regards,-Sergio A

[NZafr]

Hi Sergio,

Thank you for your reply.

Indeed with SDK 2016 R2 setting LandMarksConfiguration.maxTrackedFaces to a value bigger than 1 does not work.

I will try to install SDK 2016 R3, and let you know if it solves the issue.

Noam.

[NZafr]

Hi Sergio,

I tried to install SDK 2016 R3, but unfortunately my application does not compile with it.

Specifically, I think that the Face Tracking module is not being installed correctly.

For example, I don't see any face tracking related ".h" files added to "C:\Program Files (x86)\Intel\RSSDK\include" (".h" files are added for all other modules - I installed them all).

Therefore, it is easy to understand my my application compilation fails.

Please note that I can run the face tracking sample from the RSSDK Sample Browser without any problem.

Is that problem related to what you've mentioned about the Face Tracking sample doesn't work from Visual Studio?

Please advise.

Thanks you,

Noam.

[idata]

Hi Noam,

Thank you for your reply. It's recommended to uninstall SDK R2 before installing the R3. Did you uninstall the previous SDK? This is to avoid conflicts between the files of the SDKs. We then installed the Face tracking module and didn't have any issues with the installation.Which files do you have under "C:\Program Files (x86)\Intel\RSSDK\include"? Do you see any new files after the Face tracking module installation is complete or is the installation unsuccessful?We'll be waiting for your response.-Sergio A

[NZafr]

Hi Sergio,

I followed your installation instructions and uninstalled RSSDK 2016 R2 before installing RSSDK 2016 R3.

The installation of RSSDK 2016 R3 was successful (it was also successful when I installed it without uninstalling RSSDK 2016 R2 first).

New files were created in "C:\Program Files (x86)\Intel\RSSDK\include", but it looks like not all the required files are there.

For example: pxcfacedata.h is missing, and I think it should be there because the compilation of my application fails due to this missing file.

When I re-install RSSDK 2016 R2, pxcfacedata.h is created in "C:\Program Files (x86)\Intel\RSSDK\include", and my application compiles successfully.

Just to make things clear, the problem with pxcfacedata.h is just an example of one of the compilation issues I have with RSSDK 2016 R3, but there are more issues.

Thanks,

Noam.

[MartyG]

In 2016 R3, the SDK's features are split into optional downloadable modules for the first time. The Face module is one of these optional modules that is not in the core Essentials package. Have you downloaded and installed the Face module from the SDK downloads page?

If there are other features that you have compilation problems with, please check that you have installed all the optional modules that you may require (hand, cursor, etc).

[NZafr]

Hi Marty,

Yes, I've downloaded all the modules from the downloads page.

My problem occurs when I install only the face tracking module as well as when all the modules are installed.

Have you successfully installed and used RSSDK 2016 R3 with your application?

If yes, is pxcfacedata.h created in your "C:\Program Files (x86)\Intel\RSSDK\include"?

[MartyG]

I successfully installed and used the Face and Hand modules in the Unity game engine. I have not tested the modules with C# / C++ compilation though.

You just reminded me of a case earlier this year in which someone's program in Unity that used the Essentials and Face modules ran fine in editor mode, but would break when built into an executable. They could build an executable with the Essentials module only, but it wouldn't work when the project was built with Essentials + Face in it.

[NZafr]

Thanks for the info, Marty.

I don't know if the Unity bug is related to the C++ bug or not.

Hopefully Intel will come up with a solution.

[idata]

Hi noamz,

When the Face tracking module is installed, the files get installed to RSDDK\include\RealSense\Face. The files are named: FaceConfiguration, FaceData and FaceModule. Do you see these files after installing the module?Regards,-Sergio A

[NZafr]

Hi Sergio,

I do get the files you've mentioned in "RSDDK\include\RealSense\Face".

This means that there is no backward compatibility to RSSDK 2016 R2.

For example: all reference to pxcfacedata.h should be changed to FaceData.h

Is that correct?

[NZafr]

Comparing pxcfacedata.h (R2) and Facedata.h (R3), I see that the files are basically the same with one key difference - the types were replaced from PXC types in R2 (e.g. PXCI32) to standard C types (e.g. PXCI32 was replaced with int32_t).

This means that in order to use RSSDK 2016 R3, it is necessary to modify your application code accordingly.

[idata]

Hi noamz,

That's correct, the files in "RSDDK\include\RealSense\Face" are different in R2 and R3 and work only for that specific SDK Release. As we said earlier, we're aware that in SDK 2016 R3 the Face sample Visual Studio solution file does not work and the right team is working on this. Hopefully, a fix will be available soon.Keep us updated on your results.Sergio A

[NZafr]

Hi Sergio,

Being that RSSDK R3 is not backward compatible with R2, is there a simple way to do the transition from R2 to R3?

Let me elaborate:

Our application code is based on the face tracking sample given by Intel.

Now, obviously, if someone uses Intel's code as is (without any modifications) then the transition from R2 to R3 is simple - you simply uninstall R2, install R3, and you are ready to go.

However, although we use Intel's code as a base, we've made massive changes on it to accommodate the requirements of our application.

This means that we can't just install R3 and replace our code with the new R3 sample code.

I suppose there are a lot of RealSense users with the same problem, so I wonder if Intel have a set of instructions for making the transition from R2 to R3 for users who want to keep their code?

Thanks,

Noam.

[MartyG]

Based on the history of past SDK releases, the chances of a formal 'compatibility patch' between R2 and R3 being developed are slim to none, sadly, especially given how different R3's architecture is to previous SDKs. Similar compatibility problems were experienced in the transition from R1 to R2.

It may be worth investigating whether the R2 and R3 scripts are similar enough that you could do a Find and Replace to change the affected instructions in your code, which could speed up the editing process considerably ((e.g Find and Replace PXCI32 with int32_t).

[NZafr]

Hi Marty,

I agree, chances are slim, but it doesn't hurt trying (:

I've never used R1, so I wasn't aware that the same comparability problem was experienced in the transition from R1 to R2. Thank you for that info.

I am not familiar with the scripts you've mentioned. What are they, and where can I find them?

[MartyG]

If you mean Find And Replace, it is not a script, it is an editing function in most script editing and word-processor programs. Usually found under the Edit menu of a program, it allows you to tell it what phrase to search for, and what that exact text should be replaced with. For example:

[NZafr]

Of course I am familiar with Find-Replace.

But I think that I understand our misunderstanding.

Since you are using Unity which is a scripting language, when you say "script" you actually mean "code".

Since I use C++, when I hear "script", I think of a batch script, Python, or any other scripting language (that is often used by C++ users to perform heavy editing operation automatically).

Hence I misunderstood what you meant when you said:

"It may be worth investigating whether the R2 and R3 scripts are similar enough..."

But when I change the word "scripts" with the word "codes", it makes sense to me

[MartyG]

I apologize for the misunderstanding. So what you meant to ask was "I am not familiar with the code you mentioned" ? I read back to earlier in the discussion where Unity was mentioned. Are you asking about the TrackingAction code? To save you the trouble of installing Unity to look at it, I will attach its CS script file to this message. I will also attach the FaceTrackingRule script that TrackingAction draws face tracking configuration information from.

[NZafr]

Thanks, Marty.

Much appreciated, but I am using Face Tracking C++ code, which I am very familiar with.

Indeed a lot of find-replace work will have to be done...