Software Archive
Read-only legacy content
17065 Discussions

FaceTracking from a video



I'm able to use the FaceTracking solution from the SDK with a standard 2D webcam for detecting expression and pulse; but I would like to know if it's possible to load a recorded video (such as a .avi file for example) to process it with the SDK.


0 Kudos
23 Replies

DMITRY Z. (Intel) wrote:

Hi Hape,

You're trying to feed IR stream to Face module as color stream w/o notifying module about it. It cannot work as color and IR not aligned. SDK developers work on using IR while poor lighting condition. This may become part of future SDK release.

Hi Dimitry,
thank you for your answer! I still have some questions:
1) So is there a way to notify the face module
2) Why does the IR and color streams need to be aligned when all what I feed the face module with (through the VideoModule) is the ir stream?
(I of course also tried to only feed it with ir stream (no depth stream) and use TrackingModeType::FACE_MODE_COLOR).
3) Just for understanding, What is the difference between the IR stream and the color stream that causes this problem?

Face module algorithms are trained for color stream and not usable for IR stream without modification.


Hello everybody,

for a special scenario i want to implement a face tracking with the intel realsense camera SR300, where the camera itself is placed vertically. I'm working in Unity with the RealSense Unity Toolkit. Everything works totally fine when i have the camera horizontally aligned.

Of course the tracking breaks off if i turn the camera vertically, while using the buildIn functionality of the RealSense Unity Toolkit.

Then i found this forum post and tried to work with it. Inside the SenseToolkitManager i created a PXCMSession and also set up the PXCMVideoModule.

PXCMSession  session = PXCMSession.CreateInstance();
PXCMVideoModule videoModule;

and in OnEnable() after var faceModule = SenseManager.QueryFace(); is called i instantiated the videoModule with

videoModule = faceModule.QueryInstance<PXCMVideoModule> ();

In the Update()-function of the SenseToolkitManager, after SenseManager.AcquireFrame(true,0) is called i used the following code

PXCMCapture.Sample sample = SenseManager.QuerySample ();

sample.color.AcquireAccess (PXCMImage.Access.ACCESS_READ, PXCMImage.PixelFormat.PIXEL_FORMAT_ANY, PXCMImage.Rotation.ROTATION_ANY, PXCMImage.Option.OPTION_ANY, out data);
sample.color = session.CreateImage (, data);
sample.depth.AcquireAccess (PXCMImage.Access.ACCESS_READ, PXCMImage.PixelFormat.PIXEL_FORMAT_ANY, PXCMImage.Rotation.ROTATION_ANY, PXCMImage.Option.OPTION_ANY, out data);
sample.depth = session.CreateImage (, data);

PXCMSyncPoint sp;
pxcmStatus status = videoModule.ProcessImageAsync (sample, out sp);


if (status == pxcmStatus.PXCM_STATUS_NO_ERROR) {
	sp.Synchronize ();
        //working with the face Data
	sp.Dispose ();
_captureSample = null;


This all works fine when i use PXCMImage.Rotation.ROTATION_ANY. But at the moment when i want to use any other kind of rotation unity crashes instantly after it processes the line 

pxcmStatus status = videoModule.ProcessImageAsync (sample, out sp);

I thought it might be because of the different width and height when i rotate the image with 90 and 270 degree but it also crashes when i rotate the image with 180 degree where the width and height doesn't change.

Does anyone of you maybe see the error or the failure i made and can help me with my problem?

Thank you in advance for every good hint and help.

Best regards,