This project is a Plugin for Unreal Engine 4 of OpenFace library – OpenFace is a state-of-the art tool intended for facial landmark detection, head pose estimation, facial action unit recognition, and eye-gaze estimation made by CMU and Cambridge University.

Current developement is only available for Unreal Engine 4.18 branch in Win64 platforms

Support control objects in level using head pose estimation and gaze inside UE4, camera aquisition is made by OpenCV in running thread and can be used as movie texture inside level to provide Augmented Reality composition. Face data position and analysis can be used too but only data for now. All expose as component to be used in blueprint class.

• Screenshots:

OpenFace:

UE4 Plugin:

• Development Road-map:

  • Use landing markers data for realtime facial animation integrated as an animation blueprint of UE skeleton morphtargets;
  • Use Action units inside editor;
  • Multithread for improve aquisition frames;
  • Some filters to improve data aquisition time-series in order to apply in recorded data (2nd order lowpass, kalman and others);

• Dependency: (ThirdParty plugin directory)

  • OpenCV 3.0+
  • Boost C++ Library
  • DLib vision computing lib
  • OpenFace
  • OpenBlas
  • Intel TBB
  • NVIDIA CUDA 8+ (not included)

Plugin made by tbarbanti. Any question please contact me ([email protected])

Many thanks to TadasBaltrusaitis by the great job to bring OpenFace in github a great tool specially when the commercial solutions of facial capture is based em computer vision matlab models or depth sensors APIs and not well documented for developers and enthusiasts in general to achieve high precision.

You have to respect OpenFace, Unreal Engine, Boost, TBB, dlib, OpenBLAS, and OpenCV licenses.

OpenFace is based in following papers:

Overall system OpenFace: an open source facial behavior analysis toolkit Tadas Baltrušaitis, Peter Robinson, and Louis-Philippe Morency, in IEEE Winter Conference on Applications of Computer Vision, 2016

Facial landmark detection and tracking Constrained Local Neural Fields for robust facial landmark detection in the wild Tadas Baltrušaitis, Peter Robinson, and Louis-Philippe Morency. in IEEE Int. Conference on Computer Vision Workshops, 300 Faces in-the-Wild Challenge, 2013.

Eye gaze tracking Rendering of Eyes for Eye-Shape Registration and Gaze Estimation Erroll Wood, Tadas Baltrušaitis, Xucong Zhang, Yusuke Sugano, Peter Robinson, and Andreas Bulling in IEEE International. Conference on Computer Vision (ICCV), 2015

Facial Action Unit detection Cross-dataset learning and person-specific normalisation for automatic Action Unit detection Tadas Baltrušaitis, Marwa Mahmoud, and Peter Robinson in Facial Expression Recognition and Analysis Challenge, IEEE International Conference on Automatic Face and Gesture Recognition, 2015