This is a repository for organizing papers, codes and other resources related to the topic of Avatar (talking-face and talking-body).

If you have any suggestions (missing papers, new papers, key researchers or typos), please feel free to edit and pull a request.

• Main paper list
• Researchers list
• Toolbox for avatar
• Add paper link
• Add paper notes
• Add codes if have
• Add project page if have
• Datasets and metrics
• Related links

1. NVIDIA Research
   • Neural rendering models for human generation: vid2vid NeurIPS'18, fs-vid2vid NeurIPS'19, EG3D CVPR'22;
   • Talking-face synthesis: face-vid2vid CVPR'21, Implicit NeurIPS'22, SPACE ICCV'23, One-shot Neural Head Avatar arXiv'23;
   • Talking-body synthesis: DreamPose ICCV'23;
   • Face enhancement (relighting, restoration, etc): Lumos SIGGRAPH Asia 2022, RANA ICCV'23;
   • Authorized use of synthetic videos: Avatar Fingerprinting arXiv'23;
2. Aliaksandr Siarohin @ Snap Research
   • Neural rendering models for human generation (focus on flow-based generative models): Unsupervised-Volumetric-Animation CVPR'23, 3DAvatarGAN CVPR'23, 3D-SGAN ECCV'22, Articulated-Animation CVPR'21, Monkey-Net CVPR'19, FOMM NeurIPS'19;
3. Ziwei Liu @ Nanyang Technological University
   • Talking-face synthesis: StyleSync CVPR'23, AV-CAT SIGGRAPH Asia 2022, StyleGANX ICCV'23, StyleSwap ECCV'22, PC-AVS CVPR'21, Speech2Talking-Face IJCAI'21, VToonify SIGGRAPH Asia 2022;
   • Talking-body synthesis: MotionDiffuse arXiv'22;
   • Face enhancement (relighting, restoration, etc): Relighting4D ECCV'22;
4. Xiaodong Cun @ Tencent AI Lab:
   • Talking-face synthesis: StyleHEAT ECCV'22, VideoReTalking SIGGRAPH Asia'22, ToolTalking ICCV'23, DPE CVPR'23, CodeTalker CVPR'23, SadTalker CVPR'23;
   • Talking-body synthesis: LivelySpeaker ICCV'23;

5. Max Planck Institute for Informatics:
   • 3D face models (e.g., 3DMM): FLAME SIGGRAPH Asia 2017;

Example: [Conference'year] Title, First-author Affiliation, ProjectPage, Code

• [MM 2020] Wav2Lip: Accurately Lip-sync Videos to Any Speech, The International Institute of Islamic Thought (IIIT), India, ProjectPage, Code
• [MM 2021] Imitating Arbitrary Talking Style for Realistic Audio-Driven Talking Face Synthesis, Tsinghua University, Code,
• [CVPR 2021] Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representation, The Chinese University of Hong Kong, ProjectPage, Code
• [ICCV 2021] PIRenderer: Controllable Portrait Image Generation via Semantic Neural Rendering, Peking University, ProjectPage, Code
• [ECCV 2022] StyleHEAT: One-Shot High-Resolution Editable Talking Face Generation via Pre-trained StyleGAN, Tsinghua University, ProjectPage, Code
• [SIGGRAPH Asia 2022] VideoReTalking: Audio-based Lip Synchronization for Talking Head Video Editing In the Wild, Xidian University, ProjectPage, Code
• [AAAI 2023] DINet: Deformation Inpainting Network for Realistic Face Visually Dubbing on High Resolution Video, Virtual Human Group, Netease Fuxi AI Lab, Code
• [CVPR 2023] SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animation, Xi'an Jiaotong University, ProjectPage, Code , Note
• [CVPR 2023] DPE: Disentanglement of Pose and Expression for General Video Portrait Editing, MAIS & NLPR, Institute of Automation, Chinese Academy of Sciences, ProjectPage, Code
• [ICCV 2023] MODA: Mapping-Once Audio-driven Portrait Animation with Dual Attentions, International Digital Economy Academy (IDEA), China, ProjectPage, Code,
• [ICCV 2023] ToonTalker: Cross-Domain Face Reenactment, Tsinghua University, ProjectPage, Code
• [arXiv 2023] DreamTalk: When Expressive Talking Head Generation Meets Diffusion Probabilistic Models, Tsinghua University, ProjectPage, Code,
• [ICCV 2023] Imitator: Personalized Speech-driven 3D Facial Animation, Max Planck Institute for Intelligent Systems, ProjectPage, Code,
• [ICLR 2024] GAIA: Zero-shot Talking Avatar Generation, Microsoft Research
• [ICLR 2024 (Spotlight)] Real3D-Portrait: One-shot Realistic 3D Talking Portrait Synthesis, Zhejiang University, ProjectPage, Code,

• [ICCV 2021] AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis, University of Science and Technology of China, ProjectPage, Code
• [ECCV 2022] Learning Dynamic Facial Radiance Fields for Few-Shot Talking Head Synthesis, Tsinghua University, ProjectPage, Code
• [ICLR 2023] GeneFace: Generalized and High-Fidelity Audio-Driven 3D Talking Face Synthesis, Zhejiang University, ProjectPage, Code
• [ICCV 2023] Efficient Region-Aware Neural Radiance Fields for High-Fidelity Talking Portrait Synthesis, Beihang University, ProjectPage, Code
• [arXiv 2023] GeneFace++: Generalized and Stable Real-Time Audio-Driven 3D Talking Face Generation, Zhejiang University, ProjectPage, Code

• [arXiv 2023] Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation, Alibaba Group, ProjectPage, Code,
• [CVPR'24] MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model, National University of Singapore, ProjectPage, Code,

• [SIGGRAPH Asia 2020] Gesture Generation from Trimodal Context, Korea Advanced Institute of Science and Technology (KAIST), Code
• [ICCV 2021] Speech Drives Templates: Co-Speech Gesture Synthesis with Learned Templates, ShanghaiTech University, ProjectPage, Code
• [ICCV 2021] Audio2Motion: Generating Diverse Gestures from Speech with Conditional Variational Autoencoders, Harbin Institute of Technology, Shenzhen, Code
• [CVPR 2022] Learning Hierarchical Cross-Modal Association for Co-Speech Gesture Generation, The Chinese University of Hong Kong, ProjectPage, Code
• [CVPR 2023] Taming Diffusion Models for Audio-Driven Co-Speech Gesture Generation, The University of Hong Kong, Code

• [NeurIPS 2018] Video-to-Video Synthesis, NVIDIA, ProjectPage, Code
• [ICCV 2019] Everybody Dance Now, UC Berkeley, ProjectPage, Code

TBD

TBD

1. A general toolbox for AIGC, including common metrics and models https://github.com/open-mmlab/mmagic
2. face3d: Python tools for processing 3D face https://github.com/yfeng95/face3d
3. 3DMM model fitting using Pytorch https://github.com/ascust/3DMM-Fitting-Pytorch
4. OpenFace: a facial behavior analysis toolkit https://github.com/TadasBaltrusaitis/OpenFace
5. autocrop: Automatically detects and crops faces from batches of pictures https://github.com/leblancfg/autocrop
6. OpenPose: Real-time multi-person keypoint detection library for body, face, hands, and foot estimation https://github.com/CMU-Perceptual-Computing-Lab/openpose
7. GFPGAN: Practical Algorithm for Real-world Face Restoration https://github.com/TencentARC/GFPGAN
8. CodeFormer: Robust Blind Face Restoration https://github.com/sczhou/CodeFormer

If you are interested in avatar and digital human, we would also like to recommend you to check out other related collections:

• awesome digital human https://github.com/weihaox/awesome-digital-human