Augmented Reality: put a virtual object on the face

[Yixun_L_]

Hi,

I would like to draw a virtual object on human face.

Sine the 3D face can be obtained by the 3D scanning of RealSense F200, I can align the object with the face to get the 3D position of the object.

However, the 3D position of the object is defined in the depth camera's coordinate space. How to transfer to the color camera's coordinate space, so I can use 3rd rendering lib to draw the object after color camera calibration?

Many thanks.

EJ.

[samontab]

You can use the PXCProjection functions of the RealSense API to do that.

Here is more information about it:

https://software.intel.com/sites/landingpage/realsense/camera-sdk/v1.1/documentation/html/manuals_coordinates_mapping.html

If you need more accuracy you can of course create your own calibration and project the 3D points into the camera using an external library like OpenCV for example, but for general use the RealSense API should be more than enough.

[Yixun_L_]

Thanks a lot.

I am interested why you say the calibration has more accuracy than the PXCProjection functon?

Would you explain more?

Thanks again.

samontab wrote:

You can use the PXCProjection functions of the RealSense API to do that.

Here is more information about it:

https://software.intel.com/sites/landingpage/realsense/camera-sdk/v1.1/d...

If you need more accuracy you can of course create your own calibration and project the 3D points into the camera using an external library like OpenCV for example, but for general use the RealSense API should be more than enough.

[samontab]

You are welcome.

They provide a generic calibration that should work for all cameras. Since each camera is manufactured slightly different, it means that you can always improve that calibration using your own specific camera. This is true for other devices like the Kinect as well.

In particular, you can see that the parameters of the camera matrix of the infrared camera are a very rough estimate (the optical center is located exactly at the middle of the image for example).

[Yixun_L_]

Got it. Thanks.

1. BTW, where can I find the generic calibration tool you mentioned?

2. Can I use the chessboard along with OpenCV to do the calibration for the color camera?

For the above two methods, which one is better?

3. Is the infrared camera you mentioned the same with the color camera?

Actually, I am not quite familiar with the hardware of the Realsense camera. I think it includes a color camera and a depth camera, right?

Too many questions:-)

Thanks.

[samontab]

1.

There isn't a generic calibration tool available. The RealSense SDK contains the pre calibrated data. They use that data in their projections. A calibration is nothing more than just a couple of matrices. Getting those numbers is the tricky part though.

You can have access to some of the parameters though, like the focal length, center of projection, etc for both cameras. These are known as the intrinsic parameters.

2.

Of course you can use the OpenCV calibration. As long as you have images you can use it.

Better is a generic word. If you plan to run your app with other cameras, you should just use the RealSense API since nobody will be calibrating their own cameras. If you just want to use your own camera, you may benefit with a particular calibration with OpenCV for example.

3.

For some reason the color camera seems to have more detailed calibration parameters compared to the infrared one. Maybe they have a good reason to do that, but I don't know since it is not public.

The RealSense camera consists of two cameras and an IR projector. The IR projector emits a pattern that is read from the IR camera. That's how it gets the depth info. The color camera is just there for human consumption since it looks better with color, but it is not used at all to create the 3D structure. You may as well use a different color camera since the one included is not that great.

Take a look at this post for more information:

https://software.intel.com/en-us/forums/topic/537872

[Yixun_L_]

Thank you so much for your detailed explanation!