How to integrate the R200 with MakeHuman and Blender

[mzio]

Hello to everyone,

at the very beginning I took the decision to buy the Intel realsense R200 because I found this tutorial on youtube :

https://www.youtube.com/watch?v=QYuDpmwVIfo Integrating Faceshift with MakeHuman and Blender - YouTube

I believed that the R200 could work with Faceshift,but it does not. So,now I'm trying to find an alternative to Faceshift,possibly open source,because I want to do the same thing that it's in the tutorial : how to actively track my facial movements with the R200, project those movements on to a model created in MakeHuman, and then import any recorded animations into Blender. Thanks.

[MartyG]

After a lot of research, I concluded that trying to connect an R200 to MakeHuman would be far more trouble than it was worth, since there was no obvious ready-made way to control a non-RealSense program with the face in the same way that you can do with the hands on the F200 / SR-300 using the Touchless Controller component.

I did find this article though, that has a code sample you can download.

https://software.intel.com/en-us/articles/applying-intel-realsense-sdk-face-scans-to-a-3d-mesh%20?utm_source=ISTV&utm_medium=Video&utm_campaign=ISTV_2016 Applying Intel® RealSense™ SDK Face Scans to a 3D Mesh | Intel® Software

Whilst it does not say in the article that it works with R200, the article author has tagged 'R200 camera' in the category links at the bottom of the article. So it may be worth a look.

Ignore the link at the start of the article that points to an updated version, as that is for F200 and SR-300.

[mzio]

Thanks,but anyway,I'm not a coder at all,I don't see what to do with the code samples. I'm looking for a tool that works "out of the box",maybe like this :

https://www.youtube.com/watch?v=bVW9pEIv3is 3D photorealistic avatar from scans in less than 20 minutes - YouTube

he says : "There are two elements related to face component; 1) The face capture tool, and 2) the facial processing that transforms scans into blendshapes. for (1) there are many solutions, including what you get in the Intel RealSense SDK. For (2), we haven't finished developing it to the extent that we want (and publishing to the extent that we want), so we likely won't release it to the public until we feel satisfied with those goals"

I wonder if it works with the R200,but anyway,he put a lot of demonstration videos on the youtube and then he says that they haven't finished developing. So,what's the goal to do this ?

[MartyG]

I think we have already looked at most of the pre-made tools (Uraniom, Faceshift, ItSeez3D, etc). ItSeez3D claims to be compatible with R200.

http://itseez3d.com/blog/itseez3d-for-intel-realsense-camera-r200/ itSeez3D Scanning App for Intel® RealSense™ Camera R200 is Now Available! | itSeez3D

[mzio]

yes,and it worked,but it does not allow to actively track my facial movements with the R200 ; it does not allow to project those movements on to a model created in MakeHuman and it does not allow to import any recorded animations into Blender.

[MartyG]

You could export a model from Itseez3D that was made with R200 scanning in the OBJ model file format.

"You can export the model by emailing it as a .PLY, .OBJ or .WML model for editing in your favorite 3D Modeling or Sculpting application."

That OBJ model could then be imported into MakeHuman for editing into a good-looking character. The MakeHuman model could then be imported into Blender to have final animations added to it. The impression I get from my research though is that it is less trouble if you just skip MakeHuman and import your R200 OBJ model straight into Blender.

I know that you wanted to record the animations with your face using the R200. I feel though that the best you are going to get is to create the model scan with the R200, and will have to do the animation manually in Blender.

[mzio]

your solution does not help because faceshift produces the motion capture files of the face movements automatically,while If I use blender directly,I have to animate the face manually,with a big lost of time.

[MartyG]

Are you trying to create an animation file, or are you just trying to create a model that can be animated in real-time with your face so that you can record a non-interactive video of the animation?

[mzio]

I would like to move my face in real time and at the same time my facial movements will be projected to another character. I would like to save the video to use it directly into blender for an animation. An alternative can be these :

https://sites.google.com/a/cgspeed.com/cgspeed/motion-capture/daz-friendly-release The Daz-friendly BVH release of CMU's motion capture database - cgspeed

but I think that to use the recorded video directly can save time...

[MartyG]

Kind of like this mo-cap for Jim Henson's Word Party, except without markers. You can see the actors' moves represented on the screens behind them as roughly textured scans that are then edited later into high-detail models that the viewer sees when watching the show.

https://www.youtube.com/watch?v=5GH2YkfDroQ Word Party | Behind the Scenes Featurette | Netflix - YouTube

Even with this tech though, they do not capture the face. The body is captured, and an off-stage puppeteer controls the face parts with a remote control with joysticks.

The videogame LA Noire did use live face capture though.

https://www.youtube.com/watch?v=q2EG5J05048 L.A. Noire: "The Technology Behind Performance" - YouTube

[mzio]

possible alternatives :

https://www.youtube.com/watch?v=cQzH9E-Uegk Faceware Retargeter 5.0: Retargeting Workflow with Poses - YouTube

https://vimeo.com/120797900 Dynamixyz' Demoreel - Markeless facial motion capture with Performer Suite (Update Feb. 2015) on Vimeo

[MartyG]

Dynamixyz's licence price is $715 USD a month, so that one can probably be ruled out on the grounds of cost, as impressive as it is. Unless you have that kind of project budget, of course.

[mzio]

I'm trying to find less expensive alternatives and I found ni-mate. I'm trying to understand if it works with the R200 and I created this topic on the ni mate forum :

https://forum.ni-mate.com/t/realsense-r200-sensor-driver-download-error/819 RealSense R200 sensor driver download error - NI mate - NI mate Forum

[mzio]

http://www.3dsystems.com/store/realsense/3dme http://www.3dsystems.com/store/realsense/3dme

[idata]

Hello mario_zio,

I was wondering if you need further assistance.If you do, don't hesitate to post your questions here.Regards,Andres