Face Tracking - Remove Head Pose

[valerian_g_]

Hello to all,

I have capture landmark points on Face and I want to remove the head motion.
My goal is to keep the lips, eyes and jaw motion without the head motion.
I work with the head pose information (rotation matrix and head position center) and I do the inverse transformation.

Equation :
Vout = M^-1(Vin - T)
T = Head position center
M^-1 = Inverse rotation matrix
Vin = Point vector
Vout = New Point

This equation remove most of the motion.
When I test my result the head continue to rotate (for example, when the subject turn his head in front of the camera).

I have test with a cube with fixed position and I use this equation ( Vout = M(Vin) + T ) on it.
After, I have checked the trajectory curve of the cube against the trajectory of the raw data.
I have the same trajectory at the start but when the subject turn is head, the trajctory between the cube and the data change completly.
I don't what information, I can use to reduce this difference.

Can you help me?

Thanks

Regards,
Valerian.

[valerian_g_]

Hello,

I want to know if the data provided by the rotation matrix and the head center is reliable.

And if it is a good idea to use this data to remove the head motion.

I don't know it is possible to remove the head pose directly with the sdk or with an another method, because the inverse transformation don't work for me.

Thanks for your help.

[Tali_B_Intel]

Dear Valerian,

Thank you for your post. Basically, you seem to have taken to correct steps in your calculations. Working with coordinate systems can be quite tricky, and it is important to apply the rotation in the appropriate coordinates system. Once you’ve got one axis wrong, the entire calculation will go wrong. I’m providing you a piece of code which we use for the exact same purpose – cancelling the head rotation of a 3D landmark points vector. This code uses the 3 angles, rather than the rotation matrix, and I suggest you’ll do the same. Please note that the angles here are assumed to be in degrees.

void angles2matForGeoShape(float yaw, float pitch, float roll, float RotMat[9])

{

       Matrix<float> R;

       float fDeg2Rad = 3.14159265358979323846f/180.f;

       yaw = yaw*fDeg2Rad;

       pitch = -pitch*fDeg2Rad;

       roll = roll*fDeg2Rad;

       const float T1[9] = { 0, 0, -1, -1, 0, 0, 0, 1, 0 };

       Matrix<float> T(T1,3,3);

       const float R1[9] = { 1, 0, 0, 0, cos(roll), sin(roll), 0, -sin(roll), cos(roll) };

       const float R2[9] = { cos(pitch), 0, sin(pitch), 0, 1, 0, -sin(pitch), 0, cos(pitch) };

       const float R3[9] = { cos(yaw), -sin(yaw), 0, sin(yaw), cos(yaw), 0, 0, 0, 1 };

       // combined rotation, in object orientation coordinates

       R = T*Matrix<float>(R3,3,3).t()*Matrix<float>(R2,3,3).t()*Matrix<float>(R1,3,3).t()*T.t();

       memcpy_s(RotMat, 9*sizeof(float), R.ptr(), 9*sizeof(float));

}

After creating this rotation matrix, please repeat the calculation you suggested:  

Vout = RotMat(^-1)*(Vin – T).

Hope that helps.