megvii-research / megactor Goto Github PK

View Code? Open in Web Editor NEW

679.0 34.0 92.0 110 MB

License: Apache License 2.0

Python 99.95% Shell 0.05%

megactor's Introduction

MegActor: Harness the Power of Raw Video for Vivid Portrait Animation

Shurong Yang^*, Huadong Li^*, Juhao Wu^*, Minhao Jing^*†, Linze Li, Renhe Ji^‡, Jiajun Liang^‡, Haoqiang Fan

MEGVII Technology

^*Equal contribution ^†Lead this project ^‡Corresponding author

News & TODO List

[TODO] The code of MegActor-Sigma will be cooming soon.
[🔥🔥🔥 2024.08.28] Arxiv MegActor-Sigma paper are released.
[✨✨✨ 2024.07.02] For ease of replication, we provide a 10-minute dataset available on Google Drive, which should yield satisfactory performance..
[🔥🔥🔥 2024.06.25] Training setup released. Please refer to Training for details.
[🔥🔥🔥 2024.06.25] Integrated into OpenBayes, see the demo. Thank OpenBayes team!
[🔥🔥🔥 2024.06.17] Demo Gradio Online are released .
[🔥🔥🔥 2024.06.13] Data curation pipeline are released .
[🔥🔥🔥 2024.05.31] Arxiv MegActor paper are released.
[🔥🔥🔥 2024.05.24] Inference settings are released.

MegActor Features:

Usability: animates a portrait with video while ensuring consistent motion.

Reproducibility: fully open-source and trained on publicly available datasets.

Efficiency: ⚡200 V100 hours of training to achieve pleasant motions on portraits.

Overview

MegActor is an intermediate-representation-free portrait animator that uses the original video, rather than intermediate features, as the driving factor to generate realistic and vivid talking head videos. Specifically, we utilize two UNets: one extracts the identity and background features from the source image, while the other accurately generates and integrates motion features directly derived from the original videos. MegActor can be trained on low-quality, publicly available datasets and excels in facial expressiveness, pose diversity, subtle controllability, and visual quality.

Pre-generated results

demo.mp4

demo4.mp4

demo6.mp4

Preparation

Environments

Detailed environment settings should be found with environment.yaml

Linux

conda env create -f environment.yaml
pip install -U openmim

mim install mmengine
mim install "mmcv>=2.0.1"
mim install "mmdet>=3.1.0"
mim install "mmpose>=1.1.0"

conda install -c conda-forge cudatoolkit-dev -y

Dataset.
- For a detailed description of the data processing procedure, please refer to the accompanying below. Data Process Pipeline
- You may refer to a 10-min dataset in this format at Google Drive.
Pretrained weights

Please find our pretrained weights at https://huggingface.co/HVSiniX/RawVideoDriven. Or simply use
```
git clone https://huggingface.co/HVSiniX/RawVideoDriven && ln -s RawVideoDriven/weights weights
```

Training

We currently support two-stage training on single node machines.

Stage1(Image training):

bash train.sh train.py ./configs/train/train_stage1.yaml {number of gpus on this node}

Stage2(Video training):

bash train.sh train.py ./configs/train/train_stage2.yaml {number of gpus on this node}

Inference

Currently only single-GPU inference is supported. We highly recommend that you use --contour-preserve arg the better preserve the shape of the source face.

CUDA_VISIBLE_DEVICES=0 python eval.py --config configs/inference/inference.yaml --source {source image path} --driver {driving video path} --contour-preserve

Demo

For gradio interface, please run

python demo/run_gradio.py

BibTeX

@misc{yang2024megactorsigmaunlockingflexiblemixedmodal,
      title={MegActor-$\Sigma$: Unlocking Flexible Mixed-Modal Control in Portrait Animation with Diffusion Transformer}, 
      author={Shurong Yang and Huadong Li and Juhao Wu and Minhao Jing and Linze Li and Renhe Ji and Jiajun Liang and Haoqiang Fan and Jin Wang},
      year={2024},
      eprint={2408.14975},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2408.14975}, 
}
@misc{yang2024megactor,
      title={MegActor: Harness the Power of Raw Video for Vivid Portrait Animation}, 
      author={Shurong Yang and Huadong Li and Juhao Wu and Minhao Jing and Linze Li and Renhe Ji and Jiajun Liang and Haoqiang Fan},
      year={2024},
      eprint={2405.20851},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

Acknowledgement

Many thanks to the authors of mmengine, MagicAnimate, Controlnet_aux, and Detectron2.

Contact

If you have any questions, feel free to open an issue or contact us at [email protected], [email protected] or [email protected].

If you're seeking an internship and are interested in our work, please send your resume to [email protected] or [email protected].

Star History

megactor's People

Contributors

Stargazers

Watchers

Forkers

johndpope render-ai leemengtw yyheart zaku-zaku awekling wensiyuansix mortybug uncle9x9 xy-gao nicbair s8xy glenda1965 lycokie tufo830 tutuna molierflower sballie coder-drinker spicyguml ostwalt huiyao351 hay-man jbluv multithread3 blockeis ntt720 neo-flow e-kiss-me maigone masemxiao iam20cm n0wwa monsterdove windb3ll tel3text leeloor wuzhongdehua xupercoin johnson7788 mistyr0se fingerx jo-dean wang-kexian qinb zhangxujinsh whalefa1i dietdietdiet aishenfeng bruinxiong unlikeyly taichuai moguijoe farmingtong ro1ife kan93 billionerd bytesbot closegoingaway ytrjp paramedick fonghe vamoko yangshurong hs991023 obsidian6s eawrceqa pheagee fskeo cerviny bigcowpeking tkone2018 all164653642 macsuren cimasiro ymzhang96 promptude zytewalk z62528229 pawuket nanpusher jackzhousz elvuel linshao237 gazelei xjx777 tecworks-dev strategist922 xigua369 janmar30 jojate ingeniousfrog

megactor's Issues

数据处理要一个视频一个视频处理吗？能否提供自动话的脚本呢？

有人一起跑吗，怎么感觉效果一般？

I think，感觉跑完以后效果没有那么好，方法有点熟悉，就像是缝合了AniPortrait和EMO的story
postpoint和referencenet的desc比较熟悉 And generation deatils 的抖动有点明显
有后续有新版本可以戳一下......

Could not find library lovelive-sdk==1.82.68

如何在windows运行可否给个便携版本

ModuleNotFoundError: No module named 'densepose'

installation troubles

Hi, thanks for this great work.

I'm having lots of trouble installing the pip dependencies in conda.
Would it be possible to create a Docker version of the dependencies?

thanks

HF

If it's not too much trouble could somebody put this on HuggingFace Spaces on CPU hardware so I can duplicate and change to GPU
HF does not like env yamls i fear so a requirements text is all i really need

I did attempt this myself but ran into issues with
detectron2==0.6
and im assuming once it's fixed i would run into with
detectron2-densepose==0.6
and perhaps some of the rest

Demo不能正常运行

https://f4c5-58-240-80-18.ngrok-free.app/
这个网址貌似不能正常运行，点击运行无效果，而且视频长度是NAN，如图所示：

您好，如何优化牙齿得到更真实且连贯的牙齿

inference issue

Hi,

I'm having an issue running inference. Any ideas why this would happen?

File "/run/megactor/eval.py", line 142, in eval
pixel_values_pose = crop_move_face(control, all_face_rects, target_size=size)
File "/run/megactor/animate/utils/util.py", line 418, in crop_move_face
for j, ids in enumerate(faces['image_ids']):
TypeError: list indices must be integers or slices, not str

https://github.com/megvii-research/megactor/blob/main/animate/utils/util.py#L418

thanks!

training code

Wonderful work you have done!
May I got to know when the dataset preprocess&training code will be released?

优缺点

背景很稳定，人物面部纹理很清晰
人物轻微抖动，牙齿很乱，衣服抖动比较明显，遇到细节比较多的情况下更严重

推理卡住了，没有错误信息，显存显示占用了6G左右

按教程使用了虚拟环境安装，执行eval.py过程中卡住了，没有退出，也没有错误信息，显存显示占用了6G左右

(megactor) (base) jiangmuye@cv30:~/Projects/megactor$ CUDA_VISIBLE_DEVICES=1 python eval.py --config configs/inference/inference.yaml --source ./00001.jpg --driver ./4.mp4 --contour-preserve
/home/jiangmuye/anaconda3/envs/megactor/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py:249: FutureWarning: `AutoencoderTinyBlock` is deprecated and will be removed in version 0.29. Importing `AutoencoderTinyBlock` from `diffusers.models.unet_2d_blocks` is deprecated and this will be removed in a future version. Please use `from diffusers.models.unets.unet_2d_blocks import AutoencoderTinyBlock`, instead.
  deprecate("AutoencoderTinyBlock", "0.29", deprecation_message)
/home/jiangmuye/anaconda3/envs/megactor/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py:254: FutureWarning: `UNetMidBlock2D` is deprecated and will be removed in version 0.29. Importing `UNetMidBlock2D` from `diffusers.models.unet_2d_blocks` is deprecated and this will be removed in a future version. Please use `from diffusers.models.unets.unet_2d_blocks import UNetMidBlock2D`, instead.
  deprecate("UNetMidBlock2D", "0.29", deprecation_message)
/home/jiangmuye/anaconda3/envs/megactor/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py:259: FutureWarning: `UNetMidBlock2DCrossAttn` is deprecated and will be removed in version 0.29. Importing `UNetMidBlock2DCrossAttn` from `diffusers.models.unet_2d_blocks` is deprecated and this will be removed in a future version. Please use `from diffusers.models.unets.unet_2d_blocks import UNetMidBlock2DCrossAttn`, instead.
  deprecate("UNetMidBlock2DCrossAttn", "0.29", deprecation_message)
/home/jiangmuye/anaconda3/envs/megactor/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py:264: FutureWarning: `UNetMidBlock2DSimpleCrossAttn` is deprecated and will be removed in version 0.29. Importing `UNetMidBlock2DSimpleCrossAttn` from `diffusers.models.unet_2d_blocks` is deprecated and this will be removed in a future version. Please use `from diffusers.models.unets.unet_2d_blocks import UNetMidBlock2DSimpleCrossAttn`, instead.
  deprecate("UNetMidBlock2DSimpleCrossAttn", "0.29", deprecation_message)
/home/jiangmuye/anaconda3/envs/megactor/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py:269: FutureWarning: `AttnDownBlock2D` is deprecated and will be removed in version 0.29. Importing `AttnDownBlock2D` from `diffusers.models.unet_2d_blocks` is deprecated and this will be removed in a future version. Please use `from diffusers.models.unets.unet_2d_blocks import AttnDownBlock2D`, instead.
  deprecate("AttnDownBlock2D", "0.29", deprecation_message)
/home/jiangmuye/anaconda3/envs/megactor/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py:274: FutureWarning: `CrossAttnDownBlock2D` is deprecated and will be removed in version 0.29. Importing `AttnDownBlock2D` from `diffusers.models.unet_2d_blocks` is deprecated and this will be removed in a future version. Please use `from diffusers.models.unets.unet_2d_blocks import CrossAttnDownBlock2D`, instead.
  deprecate("CrossAttnDownBlock2D", "0.29", deprecation_message)
/home/jiangmuye/anaconda3/envs/megactor/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py:279: FutureWarning: `DownBlock2D` is deprecated and will be removed in version 0.29. Importing `DownBlock2D` from `diffusers.models.unet_2d_blocks` is deprecated and this will be removed in a future version. Please use `from diffusers.models.unets.unet_2d_blocks import DownBlock2D`, instead.
  deprecate("DownBlock2D", "0.29", deprecation_message)
/home/jiangmuye/anaconda3/envs/megactor/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py:284: FutureWarning: `AttnDownEncoderBlock2D` is deprecated and will be removed in version 0.29. Importing `AttnDownEncoderBlock2D` from `diffusers.models.unet_2d_blocks` is deprecated and this will be removed in a future version. Please use `from diffusers.models.unets.unet_2d_blocks import AttnDownEncoderBlock2D`, instead.
  deprecate("AttnDownEncoderBlock2D", "0.29", deprecation_message)
/home/jiangmuye/anaconda3/envs/megactor/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py:289: FutureWarning: `AttnSkipDownBlock2D` is deprecated and will be removed in version 0.29. Importing `AttnSkipDownBlock2D` from `diffusers.models.unet_2d_blocks` is deprecated and this will be removed in a future version. Please use `from diffusers.models.unets.unet_2d_blocks import AttnSkipDownBlock2D`, instead.
  deprecate("AttnSkipDownBlock2D", "0.29", deprecation_message)
/home/jiangmuye/anaconda3/envs/megactor/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py:294: FutureWarning: `SkipDownBlock2D` is deprecated and will be removed in version 0.29. Importing `SkipDownBlock2D` from `diffusers.models.unet_2d_blocks` is deprecated and this will be removed in a future version. Please use `from diffusers.models.unets.unet_2d_blocks import SkipDownBlock2D`, instead.
  deprecate("SkipDownBlock2D", "0.29", deprecation_message)
/home/jiangmuye/anaconda3/envs/megactor/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py:299: FutureWarning: `ResnetDownsampleBlock2D` is deprecated and will be removed in version 0.29. Importing `ResnetDownsampleBlock2D` from `diffusers.models.unet_2d_blocks` is deprecated and this will be removed in a future version. Please use `from diffusers.models.unets.unet_2d_blocks import ResnetDownsampleBlock2D`, instead.
  deprecate("ResnetDownsampleBlock2D", "0.29", deprecation_message)
/home/jiangmuye/anaconda3/envs/megactor/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py:304: FutureWarning: `SimpleCrossAttnDownBlock2D` is deprecated and will be removed in version 0.29. Importing `SimpleCrossAttnDownBlock2D` from `diffusers.models.unet_2d_blocks` is deprecated and this will be removed in a future version. Please use `from diffusers.models.unets.unet_2d_blocks import SimpleCrossAttnDownBlock2D`, instead.
  deprecate("SimpleCrossAttnDownBlock2D", "0.29", deprecation_message)
/home/jiangmuye/anaconda3/envs/megactor/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py:309: FutureWarning: `KDownBlock2D` is deprecated and will be removed in version 0.29. Importing `KDownBlock2D` from `diffusers.models.unet_2d_blocks` is deprecated and this will be removed in a future version. Please use `from diffusers.models.unets.unet_2d_blocks import KDownBlock2D`, instead.
  deprecate("KDownBlock2D", "0.29", deprecation_message)
/home/jiangmuye/anaconda3/envs/megactor/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py:314: FutureWarning: `KCrossAttnDownBlock2D` is deprecated and will be removed in version 0.29. Importing `KCrossAttnDownBlock2D` from `diffusers.models.unet_2d_blocks` is deprecated and this will be removed in a future version. Please use `from diffusers.models.unets.unet_2d_blocks import KCrossAttnDownBlock2D`, instead.
  deprecate("KCrossAttnDownBlock2D", "0.29", deprecation_message)
/home/jiangmuye/anaconda3/envs/megactor/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py:319: FutureWarning: `AttnUpBlock2D` is deprecated and will be removed in version 0.29. Importing `AttnUpBlock2D` from `diffusers.models.unet_2d_blocks` is deprecated and this will be removed in a future version. Please use `from diffusers.models.unets.unet_2d_blocks import AttnUpBlock2D`, instead.
  deprecate("AttnUpBlock2D", "0.29", deprecation_message)
/home/jiangmuye/anaconda3/envs/megactor/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py:324: FutureWarning: `CrossAttnUpBlock2D` is deprecated and will be removed in version 0.29. Importing `CrossAttnUpBlock2D` from `diffusers.models.unet_2d_blocks` is deprecated and this will be removed in a future version. Please use `from diffusers.models.unets.unet_2d_blocks import CrossAttnUpBlock2D`, instead.
  deprecate("CrossAttnUpBlock2D", "0.29", deprecation_message)
/home/jiangmuye/anaconda3/envs/megactor/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py:329: FutureWarning: `UpBlock2D` is deprecated and will be removed in version 0.29. Importing `UpBlock2D` from `diffusers.models.unet_2d_blocks` is deprecated and this will be removed in a future version. Please use `from diffusers.models.unets.unet_2d_blocks import UpBlock2D`, instead.
  deprecate("UpBlock2D", "0.29", deprecation_message)
/home/jiangmuye/anaconda3/envs/megactor/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py:334: FutureWarning: `UpDecoderBlock2D` is deprecated and will be removed in version 0.29. Importing `UpDecoderBlock2D` from `diffusers.models.unet_2d_blocks` is deprecated and this will be removed in a future version. Please use `from diffusers.models.unets.unet_2d_blocks import UpDecoderBlock2D`, instead.
  deprecate("UpDecoderBlock2D", "0.29", deprecation_message)
/home/jiangmuye/anaconda3/envs/megactor/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py:339: FutureWarning: `AttnUpDecoderBlock2D` is deprecated and will be removed in version 0.29. Importing `AttnUpDecoderBlock2D` from `diffusers.models.unet_2d_blocks` is deprecated and this will be removed in a future version. Please use `from diffusers.models.unets.unet_2d_blocks import AttnUpDecoderBlock2D`, instead.
  deprecate("AttnUpDecoderBlock2D", "0.29", deprecation_message)
/home/jiangmuye/anaconda3/envs/megactor/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py:344: FutureWarning: `AttnSkipUpBlock2D` is deprecated and will be removed in version 0.29. Importing `AttnSkipUpBlock2D` from `diffusers.models.unet_2d_blocks` is deprecated and this will be removed in a future version. Please use `from diffusers.models.unets.unet_2d_blocks import AttnSkipUpBlock2D`, instead.
  deprecate("AttnSkipUpBlock2D", "0.29", deprecation_message)
/home/jiangmuye/anaconda3/envs/megactor/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py:349: FutureWarning: `SkipUpBlock2D` is deprecated and will be removed in version 0.29. Importing `SkipUpBlock2D` from `diffusers.models.unet_2d_blocks` is deprecated and this will be removed in a future version. Please use `from diffusers.models.unets.unet_2d_blocks import SkipUpBlock2D`, instead.
  deprecate("SkipUpBlock2D", "0.29", deprecation_message)
/home/jiangmuye/anaconda3/envs/megactor/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py:354: FutureWarning: `ResnetUpsampleBlock2D` is deprecated and will be removed in version 0.29. Importing `ResnetUpsampleBlock2D` from `diffusers.models.unet_2d_blocks` is deprecated and this will be removed in a future version. Please use `from diffusers.models.unets.unet_2d_blocks import ResnetUpsampleBlock2D`, instead.
  deprecate("ResnetUpsampleBlock2D", "0.29", deprecation_message)
/home/jiangmuye/anaconda3/envs/megactor/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py:359: FutureWarning: `SimpleCrossAttnUpBlock2D` is deprecated and will be removed in version 0.29. Importing `SimpleCrossAttnUpBlock2D` from `diffusers.models.unet_2d_blocks` is deprecated and this will be removed in a future version. Please use `from diffusers.models.unets.unet_2d_blocks import SimpleCrossAttnUpBlock2D`, instead.
  deprecate("SimpleCrossAttnUpBlock2D", "0.29", deprecation_message)
/home/jiangmuye/anaconda3/envs/megactor/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py:364: FutureWarning: `KUpBlock2D` is deprecated and will be removed in version 0.29. Importing `KUpBlock2D` from `diffusers.models.unet_2d_blocks` is deprecated and this will be removed in a future version. Please use `from diffusers.models.unets.unet_2d_blocks import KUpBlock2D`, instead.
  deprecate("KUpBlock2D", "0.29", deprecation_message)
/home/jiangmuye/anaconda3/envs/megactor/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py:369: FutureWarning: `KCrossAttnUpBlock2D` is deprecated and will be removed in version 0.29. Importing `KCrossAttnUpBlock2D` from `diffusers.models.unet_2d_blocks` is deprecated and this will be removed in a future version. Please use `from diffusers.models.unets.unet_2d_blocks import KCrossAttnUpBlock2D`, instead.
  deprecate("KCrossAttnUpBlock2D", "0.29", deprecation_message)
/home/jiangmuye/anaconda3/envs/megactor/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py:375: FutureWarning: `KAttentionBlock` is deprecated and will be removed in version 0.29. Importing `KAttentionBlock` from `diffusers.models.unet_2d_blocks` is deprecated and this will be removed in a future version. Please use `from diffusers.models.unets.unet_2d_blocks import KAttentionBlock`, instead.
  deprecate("KAttentionBlock", "0.29", deprecation_message)
/home/jiangmuye/anaconda3/envs/megactor/lib/python3.10/site-packages/diffusers/models/unet_2d_condition.py:20: FutureWarning: `UNet2DConditionOutput` is deprecated and will be removed in version 0.29. Importing `UNet2DConditionOutput` from `diffusers.models.unet_2d_condition` is deprecated and this will be removed in a future version. Please use `from diffusers.models.unets.unet_2d_condition import UNet2DConditionOutput`, instead.
  deprecate("UNet2DConditionOutput", "0.29", deprecation_message)
/home/jiangmuye/anaconda3/envs/megactor/lib/python3.10/site-packages/diffusers/models/unet_2d_condition.py:25: FutureWarning: `UNet2DConditionModel` is deprecated and will be removed in version 0.29. Importing `UNet2DConditionModel` from `diffusers.models.unet_2d_condition` is deprecated and this will be removed in a future version. Please use `from diffusers.models.unets.unet_2d_condition import UNet2DConditionModel`, instead.
  deprecate("UNet2DConditionModel", "0.29", deprecation_message)
Initializing UNet MagicAnimate Pipeline...
loaded temporal unet's pretrained weights from ./weights/StableDiffusion/unet ...
### missing keys: 614; 
### unexpected keys: 0;
### Temporal Module Parameters: 417.1376 M
use appearance_encoder from ./weights/appearance_encoder
/home/jiangmuye/anaconda3/envs/megactor/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py:76: FutureWarning: `get_down_block` is deprecated and will be removed in version 0.29. Importing `get_down_block` from `diffusers.models.unet_2d_blocks` is deprecated and this will be removed in a future version. Please use `from diffusers.models.unets.unet_2d_blocks import get_down_block`, instead.
  deprecate("get_down_block", "0.29", deprecation_message)
/home/jiangmuye/anaconda3/envs/megactor/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py:213: FutureWarning: `get_up_block` is deprecated and will be removed in version 0.29. Importing `get_up_block` from `diffusers.models.unet_2d_blocks` is deprecated and this will be removed in a future version. Please use `from diffusers.models.unets.unet_2d_blocks import get_up_block`, instead.
  deprecate("get_up_block", "0.29", deprecation_message)
Concat some Reference: concat_noise_image_type is origin, reconstruct unet.conv_in
load all model from checkpoint: ./weights/checkpoint-steps9000.ckpt
global_step: 9000
load checkpoint: appearance_encoder_state_dict 662
load checkpoint: unet_state_dict 1300
load checkpoint: appearance_encoder missing keys: 0, unexpected keys: 0
load checkpoint: unet missing keys: 0, unexpected keys: 0
/home/jiangmuye/Projects/megactor/animate/unet_magic_noiseAttenST_Ada/pipeline.py:106: FutureWarning: The configuration file of this scheduler: DDIMScheduler {
  "_class_name": "DDIMScheduler",
  "_diffusers_version": "0.27.2",
  "beta_end": 0.012,
  "beta_schedule": "linear",
  "beta_start": 0.00085,
  "clip_sample": true,
  "clip_sample_range": 1.0,
  "dynamic_thresholding_ratio": 0.995,
  "num_train_timesteps": 1000,
  "prediction_type": "epsilon",
  "rescale_betas_zero_snr": false,
  "sample_max_value": 1.0,
  "set_alpha_to_one": true,
  "steps_offset": 0,
  "thresholding": false,
  "timestep_spacing": "leading",
  "trained_betas": null
}
 is outdated. `steps_offset` should be set to 1 instead of 0. Please make sure to update the config accordingly as leaving `steps_offset` might led to incorrect results in future versions. If you have downloaded this checkpoint from the Hugging Face Hub, it would be very nice if you could open a Pull request for the `scheduler/scheduler_config.json` file
  deprecate("steps_offset!=1", "1.0.0",
/home/jiangmuye/Projects/megactor/animate/unet_magic_noiseAttenST_Ada/pipeline.py:120: FutureWarning: The configuration file of this scheduler: DDIMScheduler {
  "_class_name": "DDIMScheduler",
  "_diffusers_version": "0.27.2",
  "beta_end": 0.012,
  "beta_schedule": "linear",
  "beta_start": 0.00085,
  "clip_sample": true,
  "clip_sample_range": 1.0,
  "dynamic_thresholding_ratio": 0.995,
  "num_train_timesteps": 1000,
  "prediction_type": "epsilon",
  "rescale_betas_zero_snr": false,
  "sample_max_value": 1.0,
  "set_alpha_to_one": true,
  "steps_offset": 1,
  "thresholding": false,
  "timestep_spacing": "leading",
  "trained_betas": null
}
 has not set the configuration `clip_sample`. `clip_sample` should be set to False in the configuration file. Please make sure to update the config accordingly as not setting `clip_sample` in the config might lead to incorrect results in future versions. If you have downloaded this checkpoint from the Hugging Face Hub, it would be very nice if you could open a Pull request for the `scheduler/scheduler_config.json` file
  deprecate("clip_sample not set", "1.0.0",
Initialization Done!
[2024-06-27 11:07:27,711] [INFO] [real_accelerator.py:191:get_accelerator] Setting ds_accelerator to cuda (auto detect)
Loads checkpoint by local backend from path: ./weights/aux/yolox_l_8x8_300e_coco_20211126_140236-d3bd2b23.pth
Loads checkpoint by local backend from path: ./weights/aux/dw-ll_ucoco_384.pth

请问这有可能是哪里出问题了？

huggingface上的预训练权重

按照readme里给出的命令：git clone https://huggingface.co/HVSiniX/RawVideoDriven && ln -s RawVideoDriven/weights weights
下载完之后inference的时候报错：
WARNING[XFORMERS]: xFormers can't load C++/CUDA extensions. xFormers was built for:
PyTorch 2.0.0+cu118 with CUDA 1108 (you have 1.13.0)
Python 3.10.11 (you have 3.10.13)
Please reinstall xformers (see https://github.com/facebookresearch/xformers#installing-xformers)
Memory-efficient attention, SwiGLU, sparse and more won't be available.
Set XFORMERS_MORE_DETAILS=1 for more details
/home/zx/anaconda3/envs/megactor/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py:249: FutureWarning: AutoencoderTinyBlock is deprecated and will be removed in version 0.29. Importing AutoencoderTinyBlock from diffusers.models.unet_2d_blocks is deprecated and this will be removed in a future version. Please use from diffusers.models.unets.unet_2d_blocks import AutoencoderTinyBlock, instead.
deprecate("AutoencoderTinyBlock", "0.29", deprecation_message)
/home/zx/anaconda3/envs/megactor/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py:254: FutureWarning: UNetMidBlock2D is deprecated and will be removed in version 0.29. Importing UNetMidBlock2D from diffusers.models.unet_2d_blocks is deprecated and this will be removed in a future version. Please use from diffusers.models.unets.unet_2d_blocks import UNetMidBlock2D, instead.
deprecate("UNetMidBlock2D", "0.29", deprecation_message)
/home/zx/anaconda3/envs/megactor/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py:259: FutureWarning: UNetMidBlock2DCrossAttn is deprecated and will be removed in version 0.29. Importing UNetMidBlock2DCrossAttn from diffusers.models.unet_2d_blocks is deprecated and this will be removed in a future version. Please use from diffusers.models.unets.unet_2d_blocks import UNetMidBlock2DCrossAttn, instead.
deprecate("UNetMidBlock2DCrossAttn", "0.29", deprecation_message)
/home/zx/anaconda3/envs/megactor/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py:264: FutureWarning: UNetMidBlock2DSimpleCrossAttn is deprecated and will be removed in version 0.29. Importing UNetMidBlock2DSimpleCrossAttn from diffusers.models.unet_2d_blocks is deprecated and this will be removed in a future version. Please use from diffusers.models.unets.unet_2d_blocks import UNetMidBlock2DSimpleCrossAttn, instead.
deprecate("UNetMidBlock2DSimpleCrossAttn", "0.29", deprecation_message)
/home/zx/anaconda3/envs/megactor/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py:269: FutureWarning: AttnDownBlock2D is deprecated and will be removed in version 0.29. Importing AttnDownBlock2D from diffusers.models.unet_2d_blocks is deprecated and this will be removed in a future version. Please use from diffusers.models.unets.unet_2d_blocks import AttnDownBlock2D, instead.
deprecate("AttnDownBlock2D", "0.29", deprecation_message)
/home/zx/anaconda3/envs/megactor/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py:274: FutureWarning: CrossAttnDownBlock2D is deprecated and will be removed in version 0.29. Importing AttnDownBlock2D from diffusers.models.unet_2d_blocks is deprecated and this will be removed in a future version. Please use from diffusers.models.unets.unet_2d_blocks import CrossAttnDownBlock2D, instead.
deprecate("CrossAttnDownBlock2D", "0.29", deprecation_message)
/home/zx/anaconda3/envs/megactor/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py:279: FutureWarning: DownBlock2D is deprecated and will be removed in version 0.29. Importing DownBlock2D from diffusers.models.unet_2d_blocks is deprecated and this will be removed in a future version. Please use from diffusers.models.unets.unet_2d_blocks import DownBlock2D, instead.
deprecate("DownBlock2D", "0.29", deprecation_message)
/home/zx/anaconda3/envs/megactor/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py:284: FutureWarning: AttnDownEncoderBlock2D is deprecated and will be removed in version 0.29. Importing AttnDownEncoderBlock2D from diffusers.models.unet_2d_blocks is deprecated and this will be removed in a future version. Please use from diffusers.models.unets.unet_2d_blocks import AttnDownEncoderBlock2D, instead.
deprecate("AttnDownEncoderBlock2D", "0.29", deprecation_message)
/home/zx/anaconda3/envs/megactor/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py:289: FutureWarning: AttnSkipDownBlock2D is deprecated and will be removed in version 0.29. Importing AttnSkipDownBlock2D from diffusers.models.unet_2d_blocks is deprecated and this will be removed in a future version. Please use from diffusers.models.unets.unet_2d_blocks import AttnSkipDownBlock2D, instead.
deprecate("AttnSkipDownBlock2D", "0.29", deprecation_message)
/home/zx/anaconda3/envs/megactor/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py:294: FutureWarning: SkipDownBlock2D is deprecated and will be removed in version 0.29. Importing SkipDownBlock2D from diffusers.models.unet_2d_blocks is deprecated and this will be removed in a future version. Please use from diffusers.models.unets.unet_2d_blocks import SkipDownBlock2D, instead.
deprecate("SkipDownBlock2D", "0.29", deprecation_message)
/home/zx/anaconda3/envs/megactor/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py:299: FutureWarning: ResnetDownsampleBlock2D is deprecated and will be removed in version 0.29. Importing ResnetDownsampleBlock2D from diffusers.models.unet_2d_blocks is deprecated and this will be removed in a future version. Please use from diffusers.models.unets.unet_2d_blocks import ResnetDownsampleBlock2D, instead.
deprecate("ResnetDownsampleBlock2D", "0.29", deprecation_message)
/home/zx/anaconda3/envs/megactor/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py:304: FutureWarning: SimpleCrossAttnDownBlock2D is deprecated and will be removed in version 0.29. Importing SimpleCrossAttnDownBlock2D from diffusers.models.unet_2d_blocks is deprecated and this will be removed in a future version. Please use from diffusers.models.unets.unet_2d_blocks import SimpleCrossAttnDownBlock2D, instead.
deprecate("SimpleCrossAttnDownBlock2D", "0.29", deprecation_message)
/home/zx/anaconda3/envs/megactor/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py:309: FutureWarning: KDownBlock2D is deprecated and will be removed in version 0.29. Importing KDownBlock2D from diffusers.models.unet_2d_blocks is deprecated and this will be removed in a future version. Please use from diffusers.models.unets.unet_2d_blocks import KDownBlock2D, instead.
deprecate("KDownBlock2D", "0.29", deprecation_message)
/home/zx/anaconda3/envs/megactor/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py:314: FutureWarning: KCrossAttnDownBlock2D is deprecated and will be removed in version 0.29. Importing KCrossAttnDownBlock2D from diffusers.models.unet_2d_blocks is deprecated and this will be removed in a future version. Please use from diffusers.models.unets.unet_2d_blocks import KCrossAttnDownBlock2D, instead.
deprecate("KCrossAttnDownBlock2D", "0.29", deprecation_message)
/home/zx/anaconda3/envs/megactor/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py:319: FutureWarning: AttnUpBlock2D is deprecated and will be removed in version 0.29. Importing AttnUpBlock2D from diffusers.models.unet_2d_blocks is deprecated and this will be removed in a future version. Please use from diffusers.models.unets.unet_2d_blocks import AttnUpBlock2D, instead.
deprecate("AttnUpBlock2D", "0.29", deprecation_message)
/home/zx/anaconda3/envs/megactor/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py:324: FutureWarning: CrossAttnUpBlock2D is deprecated and will be removed in version 0.29. Importing CrossAttnUpBlock2D from diffusers.models.unet_2d_blocks is deprecated and this will be removed in a future version. Please use from diffusers.models.unets.unet_2d_blocks import CrossAttnUpBlock2D, instead.
deprecate("CrossAttnUpBlock2D", "0.29", deprecation_message)
/home/zx/anaconda3/envs/megactor/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py:329: FutureWarning: UpBlock2D is deprecated and will be removed in version 0.29. Importing UpBlock2D from diffusers.models.unet_2d_blocks is deprecated and this will be removed in a future version. Please use from diffusers.models.unets.unet_2d_blocks import UpBlock2D, instead.
deprecate("UpBlock2D", "0.29", deprecation_message)
/home/zx/anaconda3/envs/megactor/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py:334: FutureWarning: UpDecoderBlock2D is deprecated and will be removed in version 0.29. Importing UpDecoderBlock2D from diffusers.models.unet_2d_blocks is deprecated and this will be removed in a future version. Please use from diffusers.models.unets.unet_2d_blocks import UpDecoderBlock2D, instead.
deprecate("UpDecoderBlock2D", "0.29", deprecation_message)
/home/zx/anaconda3/envs/megactor/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py:339: FutureWarning: AttnUpDecoderBlock2D is deprecated and will be removed in version 0.29. Importing AttnUpDecoderBlock2D from diffusers.models.unet_2d_blocks is deprecated and this will be removed in a future version. Please use from diffusers.models.unets.unet_2d_blocks import AttnUpDecoderBlock2D, instead.
deprecate("AttnUpDecoderBlock2D", "0.29", deprecation_message)
/home/zx/anaconda3/envs/megactor/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py:344: FutureWarning: AttnSkipUpBlock2D is deprecated and will be removed in version 0.29. Importing AttnSkipUpBlock2D from diffusers.models.unet_2d_blocks is deprecated and this will be removed in a future version. Please use from diffusers.models.unets.unet_2d_blocks import AttnSkipUpBlock2D, instead.
deprecate("AttnSkipUpBlock2D", "0.29", deprecation_message)
/home/zx/anaconda3/envs/megactor/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py:349: FutureWarning: SkipUpBlock2D is deprecated and will be removed in version 0.29. Importing SkipUpBlock2D from diffusers.models.unet_2d_blocks is deprecated and this will be removed in a future version. Please use from diffusers.models.unets.unet_2d_blocks import SkipUpBlock2D, instead.
deprecate("SkipUpBlock2D", "0.29", deprecation_message)
/home/zx/anaconda3/envs/megactor/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py:354: FutureWarning: ResnetUpsampleBlock2D is deprecated and will be removed in version 0.29. Importing ResnetUpsampleBlock2D from diffusers.models.unet_2d_blocks is deprecated and this will be removed in a future version. Please use from diffusers.models.unets.unet_2d_blocks import ResnetUpsampleBlock2D, instead.
deprecate("ResnetUpsampleBlock2D", "0.29", deprecation_message)
/home/zx/anaconda3/envs/megactor/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py:359: FutureWarning: SimpleCrossAttnUpBlock2D is deprecated and will be removed in version 0.29. Importing SimpleCrossAttnUpBlock2D from diffusers.models.unet_2d_blocks is deprecated and this will be removed in a future version. Please use from diffusers.models.unets.unet_2d_blocks import SimpleCrossAttnUpBlock2D, instead.
deprecate("SimpleCrossAttnUpBlock2D", "0.29", deprecation_message)
/home/zx/anaconda3/envs/megactor/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py:364: FutureWarning: KUpBlock2D is deprecated and will be removed in version 0.29. Importing KUpBlock2D from diffusers.models.unet_2d_blocks is deprecated and this will be removed in a future version. Please use from diffusers.models.unets.unet_2d_blocks import KUpBlock2D, instead.
deprecate("KUpBlock2D", "0.29", deprecation_message)
/home/zx/anaconda3/envs/megactor/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py:369: FutureWarning: KCrossAttnUpBlock2D is deprecated and will be removed in version 0.29. Importing KCrossAttnUpBlock2D from diffusers.models.unet_2d_blocks is deprecated and this will be removed in a future version. Please use from diffusers.models.unets.unet_2d_blocks import KCrossAttnUpBlock2D, instead.
deprecate("KCrossAttnUpBlock2D", "0.29", deprecation_message)
/home/zx/anaconda3/envs/megactor/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py:375: FutureWarning: KAttentionBlock is deprecated and will be removed in version 0.29. Importing KAttentionBlock from diffusers.models.unet_2d_blocks is deprecated and this will be removed in a future version. Please use from diffusers.models.unets.unet_2d_blocks import KAttentionBlock, instead.
deprecate("KAttentionBlock", "0.29", deprecation_message)
/home/zx/anaconda3/envs/megactor/lib/python3.10/site-packages/diffusers/models/unet_2d_condition.py:20: FutureWarning: UNet2DConditionOutput is deprecated and will be removed in version 0.29. Importing UNet2DConditionOutput from diffusers.models.unet_2d_condition is deprecated and this will be removed in a future version. Please use from diffusers.models.unets.unet_2d_condition import UNet2DConditionOutput, instead.
deprecate("UNet2DConditionOutput", "0.29", deprecation_message)
/home/zx/anaconda3/envs/megactor/lib/python3.10/site-packages/diffusers/models/unet_2d_condition.py:25: FutureWarning: UNet2DConditionModel is deprecated and will be removed in version 0.29. Importing UNet2DConditionModel from diffusers.models.unet_2d_condition is deprecated and this will be removed in a future version. Please use from diffusers.models.unets.unet_2d_condition import UNet2DConditionModel, instead.
deprecate("UNet2DConditionModel", "0.29", deprecation_message)
Initializing UNet MagicAnimate Pipeline...
Traceback (most recent call last):
File "/home/zx/megactor/eval.py", line 235, in
eval(args.source, args.driver,
File "/home/zx/megactor/eval.py", line 55, in eval
pipeline = MagicAnimate(config=config,
File "/home/zx/megactor/animate/unet_magic_noiseAttenST_Ada/animate.py", line 91, in init
self.text_encoder = CLIPTextModel.from_pretrained(config['pretrained_model_path'], subfolder="text_encoder")
File "/home/zx/anaconda3/envs/megactor/lib/python3.10/site-packages/transformers/modeling_utils.py", line 2629, in from_pretrained
state_dict = load_state_dict(resolved_archive_file)
File "/home/zx/anaconda3/envs/megactor/lib/python3.10/site-packages/transformers/modeling_utils.py", line 447, in load_state_dict
with safe_open(checkpoint_file, framework="pt") as f:
safetensors_rust.SafetensorError: Error while deserializing header: HeaderTooLarge
(megactor) zx@ubuntu:~/megactor$ python eval.py --config configs/inference/inference.yaml --source ./test_data/source/1.png --driver ./test_data/driver/1.mp4 --contour-preserve
WARNING[XFORMERS]: xFormers can't load C++/CUDA extensions. xFormers was built for:
PyTorch 2.0.0+cu118 with CUDA 1108 (you have 1.13.0)
Python 3.10.11 (you have 3.10.13)
Please reinstall xformers (see https://github.com/facebookresearch/xformers#installing-xformers)
Memory-efficient attention, SwiGLU, sparse and more won't be available.
Set XFORMERS_MORE_DETAILS=1 for more details
/home/zx/anaconda3/envs/megactor/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py:249: FutureWarning: AutoencoderTinyBlock is deprecated and will be removed in version 0.29. Importing AutoencoderTinyBlock from diffusers.models.unet_2d_blocks is deprecated and this will be removed in a future version. Please use from diffusers.models.unets.unet_2d_blocks import AutoencoderTinyBlock, instead.
deprecate("AutoencoderTinyBlock", "0.29", deprecation_message)
/home/zx/anaconda3/envs/megactor/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py:254: FutureWarning: UNetMidBlock2D is deprecated and will be removed in version 0.29. Importing UNetMidBlock2D from diffusers.models.unet_2d_blocks is deprecated and this will be removed in a future version. Please use from diffusers.models.unets.unet_2d_blocks import UNetMidBlock2D, instead.
deprecate("UNetMidBlock2D", "0.29", deprecation_message)
/home/zx/anaconda3/envs/megactor/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py:259: FutureWarning: UNetMidBlock2DCrossAttn is deprecated and will be removed in version 0.29. Importing UNetMidBlock2DCrossAttn from diffusers.models.unet_2d_blocks is deprecated and this will be removed in a future version. Please use from diffusers.models.unets.unet_2d_blocks import UNetMidBlock2DCrossAttn, instead.
deprecate("UNetMidBlock2DCrossAttn", "0.29", deprecation_message)
/home/zx/anaconda3/envs/megactor/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py:264: FutureWarning: UNetMidBlock2DSimpleCrossAttn is deprecated and will be removed in version 0.29. Importing UNetMidBlock2DSimpleCrossAttn from diffusers.models.unet_2d_blocks is deprecated and this will be removed in a future version. Please use from diffusers.models.unets.unet_2d_blocks import UNetMidBlock2DSimpleCrossAttn, instead.
deprecate("UNetMidBlock2DSimpleCrossAttn", "0.29", deprecation_message)
/home/zx/anaconda3/envs/megactor/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py:269: FutureWarning: AttnDownBlock2D is deprecated and will be removed in version 0.29. Importing AttnDownBlock2D from diffusers.models.unet_2d_blocks is deprecated and this will be removed in a future version. Please use from diffusers.models.unets.unet_2d_blocks import AttnDownBlock2D, instead.
deprecate("AttnDownBlock2D", "0.29", deprecation_message)
/home/zx/anaconda3/envs/megactor/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py:274: FutureWarning: CrossAttnDownBlock2D is deprecated and will be removed in version 0.29. Importing AttnDownBlock2D from diffusers.models.unet_2d_blocks is deprecated and this will be removed in a future version. Please use from diffusers.models.unets.unet_2d_blocks import CrossAttnDownBlock2D, instead.
deprecate("CrossAttnDownBlock2D", "0.29", deprecation_message)
/home/zx/anaconda3/envs/megactor/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py:279: FutureWarning: DownBlock2D is deprecated and will be removed in version 0.29. Importing DownBlock2D from diffusers.models.unet_2d_blocks is deprecated and this will be removed in a future version. Please use from diffusers.models.unets.unet_2d_blocks import DownBlock2D, instead.
deprecate("DownBlock2D", "0.29", deprecation_message)
/home/zx/anaconda3/envs/megactor/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py:284: FutureWarning: AttnDownEncoderBlock2D is deprecated and will be removed in version 0.29. Importing AttnDownEncoderBlock2D from diffusers.models.unet_2d_blocks is deprecated and this will be removed in a future version. Please use from diffusers.models.unets.unet_2d_blocks import AttnDownEncoderBlock2D, instead.
deprecate("AttnDownEncoderBlock2D", "0.29", deprecation_message)
/home/zx/anaconda3/envs/megactor/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py:289: FutureWarning: AttnSkipDownBlock2D is deprecated and will be removed in version 0.29. Importing AttnSkipDownBlock2D from diffusers.models.unet_2d_blocks is deprecated and this will be removed in a future version. Please use from diffusers.models.unets.unet_2d_blocks import AttnSkipDownBlock2D, instead.
deprecate("AttnSkipDownBlock2D", "0.29", deprecation_message)
/home/zx/anaconda3/envs/megactor/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py:294: FutureWarning: SkipDownBlock2D is deprecated and will be removed in version 0.29. Importing SkipDownBlock2D from diffusers.models.unet_2d_blocks is deprecated and this will be removed in a future version. Please use from diffusers.models.unets.unet_2d_blocks import SkipDownBlock2D, instead.
deprecate("SkipDownBlock2D", "0.29", deprecation_message)
/home/zx/anaconda3/envs/megactor/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py:299: FutureWarning: ResnetDownsampleBlock2D is deprecated and will be removed in version 0.29. Importing ResnetDownsampleBlock2D from diffusers.models.unet_2d_blocks is deprecated and this will be removed in a future version. Please use from diffusers.models.unets.unet_2d_blocks import ResnetDownsampleBlock2D, instead.
deprecate("ResnetDownsampleBlock2D", "0.29", deprecation_message)
/home/zx/anaconda3/envs/megactor/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py:304: FutureWarning: SimpleCrossAttnDownBlock2D is deprecated and will be removed in version 0.29. Importing SimpleCrossAttnDownBlock2D from diffusers.models.unet_2d_blocks is deprecated and this will be removed in a future version. Please use from diffusers.models.unets.unet_2d_blocks import SimpleCrossAttnDownBlock2D, instead.
deprecate("SimpleCrossAttnDownBlock2D", "0.29", deprecation_message)
/home/zx/anaconda3/envs/megactor/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py:309: FutureWarning: KDownBlock2D is deprecated and will be removed in version 0.29. Importing KDownBlock2D from diffusers.models.unet_2d_blocks is deprecated and this will be removed in a future version. Please use from diffusers.models.unets.unet_2d_blocks import KDownBlock2D, instead.
deprecate("KDownBlock2D", "0.29", deprecation_message)
/home/zx/anaconda3/envs/megactor/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py:314: FutureWarning: KCrossAttnDownBlock2D is deprecated and will be removed in version 0.29. Importing KCrossAttnDownBlock2D from diffusers.models.unet_2d_blocks is deprecated and this will be removed in a future version. Please use from diffusers.models.unets.unet_2d_blocks import KCrossAttnDownBlock2D, instead.
deprecate("KCrossAttnDownBlock2D", "0.29", deprecation_message)
/home/zx/anaconda3/envs/megactor/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py:319: FutureWarning: AttnUpBlock2D is deprecated and will be removed in version 0.29. Importing AttnUpBlock2D from diffusers.models.unet_2d_blocks is deprecated and this will be removed in a future version. Please use from diffusers.models.unets.unet_2d_blocks import AttnUpBlock2D, instead.
deprecate("AttnUpBlock2D", "0.29", deprecation_message)
/home/zx/anaconda3/envs/megactor/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py:324: FutureWarning: CrossAttnUpBlock2D is deprecated and will be removed in version 0.29. Importing CrossAttnUpBlock2D from diffusers.models.unet_2d_blocks is deprecated and this will be removed in a future version. Please use from diffusers.models.unets.unet_2d_blocks import CrossAttnUpBlock2D, instead.
deprecate("CrossAttnUpBlock2D", "0.29", deprecation_message)
/home/zx/anaconda3/envs/megactor/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py:329: FutureWarning: UpBlock2D is deprecated and will be removed in version 0.29. Importing UpBlock2D from diffusers.models.unet_2d_blocks is deprecated and this will be removed in a future version. Please use from diffusers.models.unets.unet_2d_blocks import UpBlock2D, instead.
deprecate("UpBlock2D", "0.29", deprecation_message)
/home/zx/anaconda3/envs/megactor/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py:334: FutureWarning: UpDecoderBlock2D is deprecated and will be removed in version 0.29. Importing UpDecoderBlock2D from diffusers.models.unet_2d_blocks is deprecated and this will be removed in a future version. Please use from diffusers.models.unets.unet_2d_blocks import UpDecoderBlock2D, instead.
deprecate("UpDecoderBlock2D", "0.29", deprecation_message)
/home/zx/anaconda3/envs/megactor/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py:339: FutureWarning: AttnUpDecoderBlock2D is deprecated and will be removed in version 0.29. Importing AttnUpDecoderBlock2D from diffusers.models.unet_2d_blocks is deprecated and this will be removed in a future version. Please use from diffusers.models.unets.unet_2d_blocks import AttnUpDecoderBlock2D, instead.
deprecate("AttnUpDecoderBlock2D", "0.29", deprecation_message)
/home/zx/anaconda3/envs/megactor/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py:344: FutureWarning: AttnSkipUpBlock2D is deprecated and will be removed in version 0.29. Importing AttnSkipUpBlock2D from diffusers.models.unet_2d_blocks is deprecated and this will be removed in a future version. Please use from diffusers.models.unets.unet_2d_blocks import AttnSkipUpBlock2D, instead.
deprecate("AttnSkipUpBlock2D", "0.29", deprecation_message)
/home/zx/anaconda3/envs/megactor/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py:349: FutureWarning: SkipUpBlock2D is deprecated and will be removed in version 0.29. Importing SkipUpBlock2D from diffusers.models.unet_2d_blocks is deprecated and this will be removed in a future version. Please use from diffusers.models.unets.unet_2d_blocks import SkipUpBlock2D, instead.
deprecate("SkipUpBlock2D", "0.29", deprecation_message)
/home/zx/anaconda3/envs/megactor/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py:354: FutureWarning: ResnetUpsampleBlock2D is deprecated and will be removed in version 0.29. Importing ResnetUpsampleBlock2D from diffusers.models.unet_2d_blocks is deprecated and this will be removed in a future version. Please use from diffusers.models.unets.unet_2d_blocks import ResnetUpsampleBlock2D, instead.
deprecate("ResnetUpsampleBlock2D", "0.29", deprecation_message)
/home/zx/anaconda3/envs/megactor/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py:359: FutureWarning: SimpleCrossAttnUpBlock2D is deprecated and will be removed in version 0.29. Importing SimpleCrossAttnUpBlock2D from diffusers.models.unet_2d_blocks is deprecated and this will be removed in a future version. Please use from diffusers.models.unets.unet_2d_blocks import SimpleCrossAttnUpBlock2D, instead.
deprecate("SimpleCrossAttnUpBlock2D", "0.29", deprecation_message)
/home/zx/anaconda3/envs/megactor/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py:364: FutureWarning: KUpBlock2D is deprecated and will be removed in version 0.29. Importing KUpBlock2D from diffusers.models.unet_2d_blocks is deprecated and this will be removed in a future version. Please use from diffusers.models.unets.unet_2d_blocks import KUpBlock2D, instead.
deprecate("KUpBlock2D", "0.29", deprecation_message)
/home/zx/anaconda3/envs/megactor/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py:369: FutureWarning: KCrossAttnUpBlock2D is deprecated and will be removed in version 0.29. Importing KCrossAttnUpBlock2D from diffusers.models.unet_2d_blocks is deprecated and this will be removed in a future version. Please use from diffusers.models.unets.unet_2d_blocks import KCrossAttnUpBlock2D, instead.
deprecate("KCrossAttnUpBlock2D", "0.29", deprecation_message)
/home/zx/anaconda3/envs/megactor/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py:375: FutureWarning: KAttentionBlock is deprecated and will be removed in version 0.29. Importing KAttentionBlock from diffusers.models.unet_2d_blocks is deprecated and this will be removed in a future version. Please use from diffusers.models.unets.unet_2d_blocks import KAttentionBlock, instead.
deprecate("KAttentionBlock", "0.29", deprecation_message)
/home/zx/anaconda3/envs/megactor/lib/python3.10/site-packages/diffusers/models/unet_2d_condition.py:20: FutureWarning: UNet2DConditionOutput is deprecated and will be removed in version 0.29. Importing UNet2DConditionOutput from diffusers.models.unet_2d_condition is deprecated and this will be removed in a future version. Please use from diffusers.models.unets.unet_2d_condition import UNet2DConditionOutput, instead.
deprecate("UNet2DConditionOutput", "0.29", deprecation_message)
/home/zx/anaconda3/envs/megactor/lib/python3.10/site-packages/diffusers/models/unet_2d_condition.py:25: FutureWarning: UNet2DConditionModel is deprecated and will be removed in version 0.29. Importing UNet2DConditionModel from diffusers.models.unet_2d_condition is deprecated and this will be removed in a future version. Please use from diffusers.models.unets.unet_2d_condition import UNet2DConditionModel, instead.
deprecate("UNet2DConditionModel", "0.29", deprecation_message)
Initializing UNet MagicAnimate Pipeline...
Traceback (most recent call last):
File "/home/zx/megactor/eval.py", line 235, in
eval(args.source, args.driver,
File "/home/zx/megactor/eval.py", line 55, in eval
pipeline = MagicAnimate(config=config,
File "/home/zx/megactor/animate/unet_magic_noiseAttenST_Ada/animate.py", line 91, in init
self.text_encoder = CLIPTextModel.from_pretrained(config['pretrained_model_path'], subfolder="text_encoder")
File "/home/zx/anaconda3/envs/megactor/lib/python3.10/site-packages/transformers/modeling_utils.py", line 2629, in from_pretrained
state_dict = load_state_dict(resolved_archive_file)
File "/home/zx/anaconda3/envs/megactor/lib/python3.10/site-packages/transformers/modeling_utils.py", line 447, in load_state_dict
with safe_open(checkpoint_file, framework="pt") as f:
safetensors_rust.SafetensorError: Error while deserializing header: HeaderTooLarge

在网上搜索的safetensors_rust.SafetensorError: Error while deserializing header: HeaderTooLarge这个错误是因为模型权重损坏，这是怎么回事啊？

An alternative project EchoMimic is open-sourced

EchoMimic is capable of generating portrait videos not only by audios and facial landmarks individually, but also by a combination of both audios and selected facial landmarks.
Project link: https://badtobest.github.io/echomimic.html
GitHub link: https://github.com/BadToBest/EchoMimic

demo_001.mp4

安装失败

在服务器上安装的时候报下面的错误，有没有人分析一下可能是什么原因？
Preparing metadata (setup.py): started
Preparing metadata (setup.py): finished with status 'error'

Pip subprocess error:
error: subprocess-exited-with-error

× python setup.py egg_info did not run successfully.
│ exit code: 1
╰─> [10 lines of output]
Traceback (most recent call last):
File "", line 2, in
File "", line 34, in
File "/tmp/pip-install-tiqmryer/deepspeed_b5c74bcf93264dae8197d11c45274a64/setup.py", line 100, in
cuda_major_ver, cuda_minor_ver = installed_cuda_version()
File "/tmp/pip-install-tiqmryer/deepspeed_b5c74bcf93264dae8197d11c45274a64/op_builder/builder.py", line 47, in installed_cuda_version
import torch.utils.cpp_extension
File "/dfs/data/miniconda/envs/megactor/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 25, in
from pkg_resources import packaging # type: ignore[attr-defined]
ImportError: cannot import name 'packaging' from 'pkg_resources' (/dfs/data/miniconda/envs/megactor/lib/python3.10/site-packages/pkg_resources/init.py)
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

failed

CondaEnvException: Pip failed

facer是什么？为什么提示没有这个模块，但是pip也不行？

No module match refile==8.0.0.post1

Hi Thank you for your amazing work. There is no matching refile==8.0.0.post1. I tried to use refile==0.4.1 and met following error:
from refile import smart_open, smart_load_from
ImportError: cannot import name 'smart_open' from 'refile' (/mnt/nfs/file_server/public/yt/miniconda3/envs/test/lib/python3.10/site-packages/refile/init.py)
Could you help me with this problem. Thank you!

Output video length doesn't sync driving video

Hi, I use your sample code and run on test_data.
However, I found that the output video is only one second, but the driving video length is 6 seconds.

1_1.mp4

Is there any fps or frame rounding setting in the code that cause this issue?

./weights/aux/densepose_model.pkl not found!

I can't find the models here

Installation issue

Why I'm getting below error, please kindly help, btw what's the name of the conda new env?
(replica) PS D:\code\MegFaceAnimate> conda env create -f environment.yaml
Collecting package metadata (repodata.json): done
Solving environment: failed

ResolvePackageNotFound:

ld_impl_linux-64==2.38=h1181459_1
tbb==2021.8.0=hdb19cb5_0
libuuid==1.41.5=h5eee18b_0
cudatoolkit-dev==11.7.0=h1de0b5d_6
mpc==1.1.0=h10f8cd9_1
gmp==6.2.1=h295c915_3
jinja2==3.1.3=py310h06a4308_0
readline==8.2=h5eee18b_0
libcufft==10.7.2.124=h4fbf590_0
xz==5.4.6=h5eee18b_0
pytorch-cuda==11.7=h778d358_5
mpfr==4.0.2=hb69a4c5_1
markupsafe==2.1.3=py310h5eee18b_0

执行安装时很多package不存在

执行安装时报如下错误，无法安装成功。基本上都是提示

WARNING: Package(s) not found: deepspeed

conda env create -f environment.yaml

# pip
detectron2==0.6【已解决】
detectron2-densepose==0.6
rrun==1.13.1
torchvision==0.15.1+cu117【换成 torchvision==0.15.1 可以安装】

其中 detectron2 已经安装成功了，参考 https://detectron2.readthedocs.io/en/latest/tutorials/install.html

python -m pip install 'git+https://github.com/facebookresearch/detectron2.git'
# (add --user if you don't have permission)

# Or, to install it from a local clone:
git clone https://github.com/facebookresearch/detectron2.git
python -m pip install -e detectron2

# On macOS, you may need to prepend the above commands with a few environment variables:
CC=clang CXX=clang++ ARCHFLAGS="-arch x86_64" python -m pip install ...

GPU memory usage for training

I have a question about the GPU memory usage for model training. I'm using a V100 32GB GPU, but I'm encountering "CUDA out of memory" errors when training for the first stage with default setting. This happens even when I set the gradient_accumulation_steps to 1. I would like to know how much VRAM is really needed for model training. I'm not sure if there's something wrong in my setup because your paper mentions that you also used V100 GPUs for training.