If you have encountered issues in Unity with an object's rotation suddenly being restricted, it may be due to a very old problem with X-Y-Z rotation called Gimbal Lock.
When you use the Rotation tool in the Unity editor, you see the object surrounded by multiple outer and inner colored rings that represent the X, Y and Z rotational axes. These are called Gimbals.
When two of these axes align parallel with each other, it can adversely affect rotation of the object. Despite the name, Gimbal Lock does not cause the rotation to freeze completely.
Gimbal Lock occurs both in 3D modeling software and mechanical devices. This Wikipedia entry explains it well.
The problem is not easy to solve, and many people have apparently tried a number of exotic solutions over the years to try to minimize the chances of it occurring. The Wiki entry states that one solution that might be appropriate to Unity users is to "...rotate one or more of the gimbals to an arbitrary position when Gimbal Lock is detected and thus reset the device."
A simpler approach is to change the Rotation Order of the gimbals in Unity - the order in which the X, Y and Z axes are prioritized when an object rotates. The order can vary from one platform to another. For example, the major 3D modeling software Maya prioritizes the order as Z, Y, X, whilst Unity seems to use the order Z, X, Y, as stated at the top of this Unity resource page:
If your objects are experiencing rotation hiccups that may possibly be related to Gimbal Lock, you can try manually changing the rotation order with a script, placing the lines for X, Y, Z in the sequential order that you want the axes to be prioritized. For instance, if you wanted Unity to rotate using the X, Y, Z order instead of the default Z, X, Y, you could try using the following C# script inside your object:
transform.eulerAngles.x = angleX; transform.eulerAngles.y = angleY; transform.eulerAngles.z = angleZ;
Yeah, the Wikipedia entry about gimbal lock mentioned using 4 axes as a better approach than 3.
Looking through the code of the TrackingAction script, it seems to use a mixed approach, using quaternion for the main rotation but eulerangles for the smoothing, constraints and inverts. So maybe this suggests that you would get a flawless 4D rotation if you had smoothing set to 0 and no constraints or inverts, but as soon as you set those extra features then you risk introducing 3D gimbal lock. Maybe.