A machine learning approach for tracking face expressions and transferring them to Unity3D and mixamo characters
In order to run the code, you will need to use matlab to generate the blendshape coefficients. The demo.m will generate the txt files shown in the folder “co”.
To generate the facial points (files located in pts), we used the DEST opensource software. The dest_align command will compute the 68 key points per face. The pose (files located in pose) were generated by openface. FaceLandmarkImg with the gaze parameter will generate the pose files. Samples are shown in pts and pose folders here.
Copy the coefficients into a Unity directory called “coeffs”. Import the “test.UnityPackage” to get the Mixamo model. View the image for the proper setup of the model and scripts.
You can see a sample by copying the text files from “co” into the “coeffs” directory.
If you use this software, please cite
Edward Kim, Christopher Moritz, “Enhancing the Communication Spectrum in Collaborative Virtual Environments”, 12th International Symposium on Visual Computing, ISVC 2016.