ieyer / deep3dfacereconstruction Goto Github PK

This is an python implement of Accurate 3D Face Reconstruction with Weakly-Supervised Learning: From Single Image to Image Set.

The method enforces a hybrid-level weakly-supervised training to achieve accurate CNN-based 3D face reconstruction.

The method reconstructs faces with high accuracy. Quantitative evaluations (shape errors in mm) on several benchmarks show its state-of-the-art performance:

The method produces high fidelity face textures meanwhile preserves identity information of input images. Scene illumination is also disentangled to guarantee a pure albedo.

The method can provide reasonable results under extreme conditions such as large pose and occlusions.

Our method aligns reconstruction faces with input images. It provides face pose estimation and 68 facial landmarks which are useful for other tasks. We conduct an experiment on AFLW_2000 dataset (NME) to evaluate the performance, as is shown in the table below:

Faces are represented with Basel Face Model 2009, which is easy for further manipulations (e.g expression transfer). ResNet-50 is used as backbone network to achieve over 50 fps (on GTX 1080) for reconstructions.

• Python >= 3.5 (numpy, scipy, pillow, opencv)
• Tensorflow >= 1.4
• Basel Face Model 2009 (BFM09)
• Expression Basis (transferred from Facewarehouse by Yudong et al.)

Optional:

• tf mesh renderer (We use it as renderer while training. Can be used at test stage too. Only on Linux.)

1. Clone the repository

git clone https://github.com/Microsoft/Deep3DFaceReconstruction
cd Deep3DFaceReconstruction

2. Download the BFM09 model and put "01_MorphableModel.mat" into ./BFM subfolder.

3. Download the Expression Basis (which is inside "CoarseData.zip"), and put "Exp_Pca.bin" into ./BFM subfolder.

4. Download the trained model at GoogleDrive, and put it into ./network subfolder.

5. Run the demo code.

python demo.py

6. To check the results, see ./output subfolder which contains:
   • "xxx.mat" : consists of cropped input image, corresponding 5p and 68p landmarks, and output coefficients of R-Net.
   • "xxx_mesh.obj" : 3D face mesh in canonical view (best viewed in MeshLab).

1. The model is trained without augmentation so that a pre-alignment with 5 facial landmarks is necessary. We put some examples in the ./input subfolder for reference.

2. Current model is trained under the assumption of 3-channel scene illumination (instead of monochromatic lights described in the paper).

3. If you have any questions, please contact Yu Deng ([email protected]) or Jiaolong Yang ([email protected]).

Please cite the following paper if this model helps your research:

@misc{deng2019accurate,
    title={Accurate 3D Face Reconstruction with Weakly-Supervised Learning: From Single Image to Image Set},
    author={Yu Deng and Jiaolong Yang and Sicheng Xu and Dong Chen and Yunde Jia and Xin Tong},
    year={2019},
    eprint={1903.08527},
    archivePrefix={arXiv},
    primaryClass={cs.CV}
}