Kizuna-Ai MMD demo : face capture via single RGB camera

• Python 3.7+
• Linux, Windows or macOS
• mxnet (>=1.4)
• node.js and npm or yarn

While not required, for optimal performance(especially for the detector) it is highly recommended to run the code using a CUDA enabled GPU.

• node ./NodeServer/server.js
• make -C ./PythonClient/rcnn/
• python3 ./PythonClient/vtuber_usb_camera.py --gpu -1

RetinaFace: Single-stage Dense Face Localisation in the Wild of CVPR 2020, is a practical single-stage SOTA face detector. It is highly recommended to read the official repo RetinaFace (mxnet version).

However, since the detection target of the face capture system is in the middle-close range, there is no need for complex pyramid scaling. We designed and published Faster RetinaFace to trade off between speed and accuracy, which can reach 500~1000 fps on normal laptops.

The 2D pre-trained model is from the deep-face-alignment repository, based on the heatmap regression face alignment algorithm at TPAMI 2019. The model is trained on i-bug 300-W datasets, which annotated as below:

• head-pose-estimation

3D Gaze Estimation is based on

and the head posed

• Laser Eye : Gaze Estimation via Deep Neural Networks

• threejs.org: Applying Three.js WebGL Loader to render MMD models on web pages.
• kizunaai.com: モデルは無料でご利用いただけます.

@misc{sun2020backbone,
      title={A Backbone Replaceable Fine-tuning Network for Stable Face Alignment},
      author={Xu Sun and Yingjie Guo and Shihong Xia},
      year={2020},
      eprint={2010.09501},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

@article{Bulat2018Hierarchical,
      title={Hierarchical binary CNNs for landmark localization with limited resources},
      author={Bulat, Adrian and Tzimiropoulos, Yorgos},
      journal={IEEE Transactions on Pattern Analysis & Machine Intelligence},
      year={2018},
}

@InProceedings{Deng_2020_CVPR,
      author = {Deng, Jiankang and Guo, Jia and Ververas, Evangelos and Kotsia, Irene and Zafeiriou, Stefanos},
      title = {RetinaFace: Single-Shot Multi-Level Face Localisation in the Wild},
      booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
      month = {June},
      year = {2020}
}