Tracking faces as fast as lightning!
mkdir -p assets/EMOCA
wget https://download.is.tue.mpg.de/emoca/assets/EMOCA/models/EMOCA_v2_lr_mse_20.zip -O assets/EMOCA_v2_lr_mse_20.zip
unzip assets/EMOCA_v2_lr_mse_20.zip -d assets/EMOCA
rm assets/EMOCA_v2_lr_mse_20.zip
wget --post-data "username=[username]&password=[password]" https://download.is.tue.mpg.de/download.php\?domain\=flame\&sfile\=FLAME2020.zip\&resume\=1 -O 'assets/FLAME2020.zip' --no-check-certificate
wget --post-data "username=[username]&password=[password]" https://download.is.tue.mpg.de/download.php\?domain\=flame\&resume\=1\&sfile\=TextureSpace.zip -O 'assets/TextureSpace.zip' --no-check-certificate
mkdir -p assets/FLAME
unzip assets/FLAME2020.zip -d assets/FLAME
rm assets/FLAME2020.zip
mv assets/FLAME/Readme.pdf assets/FLAME/Readme_FLAME.pdf
unzip assets/TextureSpace.zip -d assets/FLAME
rm assets/TextureSpace.zip
mv assets/FLAME/Readme.pdf assets/FLAME/Readme_Texture.pdf
unzip assets/FLAME_embedding.zip -d assets/FLAME
- python >= 3.8
- pytorch == 2.0
- pytorch3d == 0.7.3 (Following PyTorch3D)
- mediapipe
- face_alignment
- ffmpeg >= 5.1 (conda install -c conda-forge ffmpeg or Static Build)
- pip install tqdm rich lmdb colored pykalman av pytorch-lightning chumpy
Our tracker firstly tracks the expression and pose of the target head based on EMOCA v2, then optimize the camera pose based on landmarks (face_alignment and mediapipe).
Please refer to the following command to run the tracker.
python track_lightning.py -d 7 --data ./assets/demo.mp4 -v --synthesis
- The camera is calibrated based on 32 randomly sampled frames.
- The running speed is greatly affected by the iteration number of landmark optimization.
- Tracking performance is greatly affected by face_alignment and landmarks detected by mediapipe.