Kizuna-Ai MMD demo : face capture via single RGB camera
- Python 3.7+
- Linux, Windows or macOS
- mxnet (>=1.4)
- node.js and npm or yarn
While not required, for optimal performance(especially for the detector) it is highly recommended to run the code using a CUDA enabled GPU.
node ./NodeServer/server.js
make -C ./PythonClient/rcnn/
python3 ./PythonClient/vtuber_usb_camera.py --gpu -1
RetinaFace: Single-stage Dense Face Localisation in the Wild of CVPR 2020, is a practical single-stage SOTA face detector. It is highly recommended to read the official repo RetinaFace (mxnet version).
However, since the detection target of the face capture system is in the middle-close range, there is no need for complex pyramid scaling. We designed and published Faster RetinaFace to trade off between speed and accuracy, which can reach 500~1000 fps on normal laptops.
Plan | Inference | Postprocess | Throughput Capacity (FPS) |
---|---|---|---|
9750HQ+1660TI | 0.9ms | 1.5ms | 500~1000 |
Jetson-Nano | 4.6ms | 11.4ms | 80~200 |
The 2D pre-trained model is from the deep-face-alignment repository, based on the heatmap regression face alignment algorithm at TPAMI 2019. The model is trained on i-bug 300-W datasets, which annotated as below:
3D Gaze Estimation is based on
and the head posed
- threejs.org: Applying Three.js WebGL Loader to render MMD models on web pages.
- kizunaai.com: モデルは無料でご利用いただけます.
@misc{sun2020backbone,
title={A Backbone Replaceable Fine-tuning Network for Stable Face Alignment},
author={Xu Sun and Yingjie Guo and Shihong Xia},
year={2020},
eprint={2010.09501},
archivePrefix={arXiv},
primaryClass={cs.CV}
}
@article{Bulat2018Hierarchical,
title={Hierarchical binary CNNs for landmark localization with limited resources},
author={Bulat, Adrian and Tzimiropoulos, Yorgos},
journal={IEEE Transactions on Pattern Analysis & Machine Intelligence},
year={2018},
}
@InProceedings{Deng_2020_CVPR,
author = {Deng, Jiankang and Guo, Jia and Ververas, Evangelos and Kotsia, Irene and Zafeiriou, Stefanos},
title = {RetinaFace: Single-Shot Multi-Level Face Localisation in the Wild},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2020}
}