qiulin-w / safa Goto Github PK

I have downloaded face-parsing.PyTorch, however it attempted to import Resnet18, which is not possible in the publicly available PyTorch versions. The proper way to import is:

from resnet_pytorch import ResNet model = ResNet.from_pretrained('resnet18', num_classes=10)

The following error is thrown if attempting to use pytorch-resnet:

File "/SAFA/face_parsing/model.py", line 112, in forward feat8, feat16, feat32 = self.resnet(x) ValueError: not enough values to unpack (expected 3, got 1)

The face-parsing project does not indicate how Resnet has been implemented. How have you installed it on your system?

tar: It does not appear to be a tar archive

please add a google colab for inference

Hello,I'd like to ask if I can use picture as driving_input instead of video to obtain the output results of pictures.

First, I want to congratulate you on an ingenious paper. The idea of using FLAME with face alignment and masking is simply brilliant. I am getting very good results with your current code. However, I would like to ask if you could provide some brief advice on how to raise the resolution (e.g. from 256x256 to 512x512). I attempted to adjust settings in the code, but there were persistent tensor mismatch problems. I presume the dataset must be re-trained at higher resolution? Please advise, if you will have time.

Could you release the code that computes these metrics？

When I execute the following code, I had the bellow error.
How we execute the "animation_demo.py" without error?

! python animation_demo.py --config config/end2end.yaml --checkpoint path/to/checkpoint --source_image_pth path/to/source_image --driving_video_pth path/to/driving_video --relative --adapt_scale --find_best_frame

Traceback (most recent call last):
File "animation_demo.py", line 18, in
from modules.tdmm_estimator import TDMMEstimator
File "/content/SAFA/modules/tdmm_estimator.py", line 7, in
from pytorch3d.io import load_obj
File "/usr/local/lib/python3.7/dist-packages/pytorch3d/io/init.py", line 4, in
from .obj_io import load_obj, load_objs_as_meshes, save_obj
File "/usr/local/lib/python3.7/dist-packages/pytorch3d/io/obj_io.py", line 14, in
from pytorch3d.renderer import TexturesAtlas, TexturesUV
File "/usr/local/lib/python3.7/dist-packages/pytorch3d/renderer/init.py", line 3, in
from .blending import (
File "/usr/local/lib/python3.7/dist-packages/pytorch3d/renderer/blending.py", line 9, in
from pytorch3d import _C
ImportError: /usr/local/lib/python3.7/dist-packages/pytorch3d/_C.cpython-37m-x86_64-linux-gnu.so: undefined symbol: _ZN3c104impl23ExcludeDispatchKeyGuardC1ENS_11DispatchKeyE

python animation_demo.py --config config/end2end.yaml --checkpoint ./ckpt/final_3DV.tar --source_image_pth ./assets/EM.jpeg --driving_video_pth ./assets/02.mp4 --relative --adapt_scale --find_best_frame --cpu gives me:

/Users/user/miniconda3/envs/safa3/lib/python3.7/site-packages/skimage/transform/_warps.py:105: UserWarning: The default mode, 'constant', will be changed to 'reflect' in skimage 0.15.
  warn("The default mode, 'constant', will be changed to 'reflect' in "
/Users/user/miniconda3/envs/safa3/lib/python3.7/site-packages/skimage/transform/_warps.py:110: UserWarning: Anti-aliasing will be enabled by default in skimage 0.15 to avoid aliasing artifacts when down-sampling images.
  warn("Anti-aliasing will be enabled by default in skimage 0.15 to "
animation_demo.py:32: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  config = yaml.load(f)
blend_scale:  1
/Users/user/miniconda3/envs/safa3/lib/python3.7/site-packages/torch/functional.py:478: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at  /Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/native/TensorShape.cpp:2895.)
  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
/Users/user/miniconda3/envs/safa3/lib/python3.7/site-packages/torchvision/models/_utils.py:209: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and will be removed in 0.15, please use 'weights' instead.
  f"The parameter '{pretrained_param}' is deprecated since 0.13 and will be removed in 0.15, "
/Users/user/miniconda3/envs/safa3/lib/python3.7/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and will be removed in 0.15. The current behavior is equivalent to passing `weights=MobileNet_V2_Weights.IMAGENET1K_V1`. You can also use `weights=MobileNet_V2_Weights.DEFAULT` to get the most up-to-date weights.
  warnings.warn(msg)
creating the FLAME Decoder
/Users/user/miniconda3/envs/safa3/lib/python3.7/site-packages/pytorch3d/io/obj_io.py:533: UserWarning: Mtl file does not exist: ./modules/data/template.mtl
  warnings.warn(f"Mtl file does not exist: {f}")
[W NNPACK.cpp:51] Could not initialize NNPACK! Reason: Unsupported hardware.
128it [03:03,  1.48s/it]
Best frame: 120
/Users/user/miniconda3/envs/safa3/lib/python3.7/site-packages/torch/nn/functional.py:4216: UserWarning: Default grid_sample and affine_grid behavior has changed to align_corners=False since 1.3.0. Please specify align_corners=True if the old behavior is desired. See the documentation of grid_sample for details.
  "Default grid_sample and affine_grid behavior has changed "
Traceback (most recent call last):
  File "animation_demo.py", line 216, in <module>
    relative=opt.relative, adapt_movement_scale=opt.adapt_scale, cpu=opt.cpu)
  File "animation_demo.py", line 83, in make_animation
    driving_initial = driving[:, :, 0].cuda()
  File "/Users/user/miniconda3/envs/safa3/lib/python3.7/site-packages/torch/cuda/__init__.py", line 211, in _lazy_init
    raise AssertionError("Torch not compiled with CUDA enabled")
AssertionError: Torch not compiled with CUDA enabled

it seems to come from the animation_demo.py file with the absence of the if not cpu condition in line 83.

How can I modify the file to solve this properly?

Hi, would you be interested in sharing a web demo on Huggingface Spaces for SAFA?

It would make this model more accessible as it would allow people to try out the model directly from the browser. Some other recent machine learning model repos have set up Spaces for easy access:

github: https://github.com/salesforce/BLIP
Spaces: https://huggingface.co/spaces/akhaliq/BLIP

github: https://github.com/facebookresearch/omnivore
Spaces: https://huggingface.co/spaces/akhaliq/omnivore

Spaces is completely free, and I can help setup a Gradio Space. Here are some getting started instructions if you'd prefer to do it yourself: https://huggingface.co/blog/gradio-spaces

Is there a way to keep the audio in the video generated with animation_demo.py? Many thanks.

Dear authors,
Thanks for sharing such a wonderful and interesting project! May I ask if you could release the pre-trained 3DMM model? i.e., the checkpoint after running the 3DMM Estimator Pre-training step with code

CUDA_VISIBLE_DEVICES="0,1,2,3" python -m torch.distributed.launch --nproc_per_node 4 run_ddp.py --config config/pretrain.yaml

As I'd like to run your end-to-end training step with a fully pre-trained 3DMM model given by "--tdmm_checkpoint".

Looking forward to your reply. Thanks in advance!

Did the author forget to upload it? Or is there another way to download?

Prior to generating the output video, I would like to log the normal maps for each frame, if possible. I see that normal maps are being generated in the tdmm_estimator prior to generation and stored in render_ops, but what I need is a 2nd pass normal map for the final, deformed source face in the generator.

In class OcclusionAwareGenerator, could you suggest where is the final data array from which the normal map could be derived, or should it be re-calculated in some way? I haven't had much time to dig into the code, and am hoping that you will immediately know what I am talking about and provide some quick pointers.

Did maybe somebody got SAFA running with a Nvidia GeForce 30xx?
As far as I know this series requires Cuda 11.x.
Currently I experienced a variety of different errors when trying to install this module.

If you had some success, could you please post the output of "pip list" or provide some instructions / hints?

Thanks

Hello. I love your great work.

I have a question.

When I tried to pre-train 3DMM estimator, I found 'param loss' (the line 305 in model.py)

But, I couldn't understand the role of this loss term...

Could you explain more about this loss term? (How param loss work)

For 3DMM parameters (Shape, Expression), how can we regulate this parameters although there are no GT (Ground Truth). (I understand 'ldmk_loss' because we prepare ldmk GT before training)

Please forgive me

Thank you.

I have tested some data collected by myself.
the results are terrible.
could you provide some test data showed in your readme?

I download SAFA model checkpoint from Google Drive,and use data.pkl to the checkpoint.But there are some errors.
File "/home/mpl/anaconda3/envs/py37/lib/python3.7/site-packages/torch/serialization.py", line 764, in _legacy_load
magic_number = pickle_module.load(f, **pickle_load_args)
_pickle.UnpicklingError: A load persistent id instruction was encountered,
but no persistent_load function was specified.