janaldochen / anim-nerf Goto Github PK

Thanks for sharing this wonderful work!
As you mentioned in the paper,
"
We set the hyper-parameters as |N (𝑖)| = 4,
𝛿 = 0.2, 𝜆1 = 0.001, 𝜆2 = 0.01 and 𝜆𝑑 = 0.1. We use 512×512
image in all experiments. For training the model, we adopt the
Adam optimizer [52], and it spends about 13 hours on 2 Nvidia
GeForce RTX 3090 24GB GPUs
"
can you specify what the parameter is in the config file?
|N (𝑖)| : batch_size
𝛿 : dis_threshold
𝜆1 : lambda_pose in opt.py
𝜆2 : lambda_pose_smooth in opt.py
𝜆𝑑 : lambda_background

Is this right variables?

How to estimate the camera intrinsics K ? In your paper, the camera intrinsics is estimated, but the people snapsshot dataset provide a camera file including camera intrinsics.

Hi, thanks for your great work.
I'm new in this field, I want to ask a simple question: Is that possible to drive the avatar move and do some actions? And if so, how can I make it ? thank U !

Hi folks, @JanaldoChen @woo1 et al.,

I tried the demo for in-the-wild videos and found fittings seem smaller than expected. May I check if this is a common issue?

I attached an example here,

Hi @JanaldoChen @woo1 et al.,

May I ask if we intend to empirically use a slightly smaller focal len which is assumed to be closer to the in-the-wild deployment? Thx & rgds,

Thanks for your wonderful work!
I wonder if the input video only captures half of a body (e.g. upper torso), will Anim-NeRF reconstruct that part well?
For example, the image below.

Thank you for your great work
assuming i trained a model and now i want to continue the training process
so i run train.py file and changed the following :

train_resume = True
train_ckpt_path = "checkpoints/XXX/last.ckpt"

i saw there are two more params to set :
model_names_to_load = ?
pretrained_model_requires_grad = ?

what should be the value of them ?
there is any thing else i need to change?

thanks

Thanks for the great work! TorchMetrics has renamed ssim-> scale_invariant_signal_noise_ratio and psnr-> peak_signal_noise_ratio. The old names are being deprecated in v0.7 and will be removed in v0.8.

Lightning-AI/torchmetrics@8e9eb8e

Maybe the imports and usages can also be update to the new name?

Hi, in novel_pose.py you are using mocap information from mixamo file. Can you share that file and make it a configurable path?

Hi, thanks for sharing this wonderful work! In the currently released code, the people_snapshot dataset already has the associated smpl and camera parameters, which is easy to get started with. Could you further give some instruction on how to prepare these necessary inputs for a custom monocular video (assuming this video is similar to those in people_snapshot)? Something like a python -m tools.custom_video --data_root ${path_to_custom_video} along with a python -m tools.prepare_template_for_custom_video would be much appreciated.

would be great to test it out on custom videos via colab.
also , is the green screen a must?

Hello, thank you for sharing your work.

I have a question about whether can I use my own video data to test on your model ?

Another question is how did you extract the colorized texture from 2D image and apply it on 3D obj ? My goal is about extract the texture from 2D image and apply it on 3D SMPL model. Could you give me some advice about that?

Thanks.

Hello,thank you for making such wonderful project,and I want to know how to compute the evaluation index such as P2S and Chamfer distance,I would be appreaciate it if you can tell me,thank you!

Hi, thanks for sharing your excellent work.

Would you please share other pre-trained models for the data in People Snapshot?

Best wishes!

Hi, Thanks for sharing this wonderful code!
When I try training with this command.
python train.py --cfg_file configs/people_snapshot/male-3-casual.yml
this error raises

Traceback (most recent call last):
File "train.py", line 425, in
cfg = get_cfg()
File "/Anim-NeRF/config.py", line 105, in get_cfg
cfg = get_default_config()
File "/Anim-NeRF/config.py", line 52, in get_default_config
cfg.train.cam_IDs = None
File "/Anim-NeRF/venv/lib/python3.7/site-packages/yacs/config.py", line 158, in setattr
type(value), name, _VALID_TYPES
File "/Anim-NeRF/venv/lib/python3.7/site-packages/yacs/config.py", line 521, in _assert_with_logging
assert cond, msg
AssertionError: Invalid type <class 'NoneType'> for key cam_IDs; valid types = {<class 'bool'>, <class 'float'>, <class 'tuple'>, <class 'int'>, <class 'list'>, <class 'str'>}

This error did not raised yesterday.
How can I fix it?

Command:
!python -m tools.prepare_template --data_root data/custom --people_ID BAR_APose --model_type smpl --gender neutral --model_path smplx/models/

Result:
0% 0/256 [00:00<?, ?it/s]
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/content/drive/MyDrive/ColabNotebooks/NeRF/Anim-NeRF/tools/prepare_template.py", line 86, in
distances_chunk = body_mesh.nearest.signed_distance(points_chunk)
File "/usr/local/lib/python3.7/dist-packages/trimesh/proximity.py", line 307, in signed_distance
return signed_distance(self._mesh, points)
File "/usr/local/lib/python3.7/dist-packages/trimesh/proximity.py", line 231, in signed_distance
closest, distance, triangle_id = closest_point(mesh, points)
File "/usr/local/lib/python3.7/dist-packages/trimesh/proximity.py", line 138, in closest_point
candidates = nearby_faces(mesh, points)
File "/usr/local/lib/python3.7/dist-packages/trimesh/proximity.py", line 43, in nearby_faces
rtree = mesh.triangles_tree
File "/usr/local/lib/python3.7/dist-packages/trimesh/caching.py", line 103, in get_cached
value = function(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/trimesh/base.py", line 811, in triangles_tree
tree = triangles.bounds_tree(self.triangles)
File "/usr/local/lib/python3.7/dist-packages/trimesh/triangles.py", line 338, in bounds_tree
tree = util.bounds_tree(triangle_bounds)
File "/usr/local/lib/python3.7/dist-packages/trimesh/util.py", line 1597, in bounds_tree
import rtree
ModuleNotFoundError: No module named 'rtree'

I used these command of Visualization:
python novel_view.py --ckpt_path checkpoints/male-3-casual/last.ckpt
python extract_mesh.py --ckpt_path checkpoints/male-3-casual/last.ckpt
python novel_view.py --ckpt_path checkpoints/male-3-casual/last.ckpt --betas_2th -2.0
python novel_view.py --ckpt_path checkpoints/male-3-casual/last.ckpt --betas_2th 3.0
python novel_pose.py --ckpt_path checkpoints/male-3-casual/last.ckpt
Both got the same error report. And they can't work.

The error report both like :

(Anim-NeRF) near@near:~/Github/Anim-NeRF$ python novel_view.py --ckpt_path checkpoints/female-3-casual/last.ckpt --betas_2th -2.0
WARNING: You are using a SMPL model, with only 10 shape coefficients.
Setting up [LPIPS] perceptual loss: trunk [alex], v[0.1], spatial [off]
Loading model from: /home/near/miniconda3/envs/Anim-NeRF/lib/python3.8/site-packages/lpips/weights/v0.1/alex.pth
Traceback (most recent call last):
File "novel_view.py", line 146, in
system = AnimNeRFSystem.load_from_checkpoint(args.ckpt_path).to(device)
File "/home/near/miniconda3/envs/Anim-NeRF/lib/python3.8/site-packages/pytorch_lightning/core/saving.py", line 156, in load_from_checkpoint
model = cls._load_model_state(checkpoint, strict=strict, **kwargs)
File "/home/near/miniconda3/envs/Anim-NeRF/lib/python3.8/site-packages/pytorch_lightning/core/saving.py", line 204, in _load_model_state
keys = model.load_state_dict(checkpoint["state_dict"], strict=strict)
File "/home/near/miniconda3/envs/Anim-NeRF/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1667, in load_state_dict
raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for AnimNeRFSystem:
Missing key(s) in state_dict: "evaluator.lpips.scaling_layer.shift", "evaluator.lpips.scaling_layer.scale", "evaluator.lpips.net.slice1.0.weight", "evaluator.lpips.net.slice1.0.bias", "evaluator.lpips.net.slice2.3.weight", "evaluator.lpips.net.slice2.3.bias", "evaluator.lpips.net.slice3.6.weight", "evaluator.lpips.net.slice3.6.bias", "evaluator.lpips.net.slice4.8.weight", "evaluator.lpips.net.slice4.8.bias", "evaluator.lpips.net.slice5.10.weight", "evaluator.lpips.net.slice5.10.bias", "evaluator.lpips.lin0.model.1.weight", "evaluator.lpips.lin1.model.1.weight", "evaluator.lpips.lin2.model.1.weight", "evaluator.lpips.lin3.model.1.weight", "evaluator.lpips.lin4.model.1.weight", "evaluator.lpips.lins.0.model.1.weight", "evaluator.lpips.lins.1.model.1.weight", "evaluator.lpips.lins.2.model.1.weight", "evaluator.lpips.lins.3.model.1.weight", "evaluator.lpips.lins.4.model.1.weight".

Could you please give me some advice to solve this problem ?
Thanks a lot !

Thank you for your great work. I have a question whether the code can generate a textured 3D model. I ran your code and did not generate a textured 3D model.

Hey,just like the title.Is the output model watertight?If not,any way to fix it?

I noticed that the pose smooth loss is commented out. Does it mean there is no pose refinement? Or adding pose refinement performs worse? Thanks!

Dear Jianchuan,

thanks for this great work and updating the code for training on customized video.

I wondered if you could also provide some script to test on novel pose synthesis on IPER or customized testing video.
Or, could you give some general guidance to perform novel pose synthesis on a customized testing video?

Thanks!

Thanks for your great work!
For novel pose synthesis, I note that all pose cases obtained from mixamo, which provides 'fbx' animation files. How do you parse smpl params (result.pkl) from the fbx file? or you directly extract smpl params from accroding videos via VIBE?

Hi @JanaldoChen :
I see you changed some code in smplx to support for SMPL models, which is great. But I wonder if the code of transformation from observation space to canonical space support for MANO hand models? Particularly for the code of clac_ober2cano_transform() and get_neighbs(), do the support for smplx in this repo?

Thank you for your work again!

Thanks for the human project! When will the training code released ?

Installed all requirements.
Downloaded the pre-trained model and put into /checkpoints from Google Drive.
Downloaded the people-snapshot dataset (male-3-casual), unzipped and put into /data from Google Drive.
Run extract mesh inference as follows
!python extract_mesh.py --ckpt_path checkpoints/male-3-casual/last.ckpt

Got error:

Traceback (most recent call last):
File "extract_mesh.py", line 103, in
system = AnimNeRFSystem.load_from_checkpoint(args.ckpt_path).to(device)
File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/core/saving.py", line 156, in load_from_checkpoint
model = cls._load_model_state(checkpoint, strict=strict, kwargs)
File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/core/saving.py", line 198, in _load_model_state
model = cls(_cls_kwargs)
File "/content/drive/MyDrive/ColabNotebooks/NeRF/Anim-NeRF/train.py", line 125, in init
query_inside=self.hparams.query_inside,
File "/content/drive/MyDrive/ColabNotebooks/NeRF/Anim-NeRF/models/anim_nerf.py", line 79, in init
self.body_model = create(model_path, model_type, gender=gender)
File "/content/drive/MyDrive/ColabNotebooks/NeRF/Anim-NeRF/smplx/body_models.py", line 2457, in create
raise ValueError(f'Unknown model type {model_type}, exiting!')
ValueError: Unknown model type models, exiting!

Hi. First, I'm very happy for your work. Thanks for sharing a nice work.

By the way, I'm not familiar with pytorch-lighting package but there is a problem with loading your pre-trained checkpoint checkpoints/male-3-casual/last.ckpt in this line:

system = AnimNeRFSystem.load_from_checkpoint(args.ckpt_path).to(device)

The error is like this:

Error(s) in loading state_dict for AnimNeRFSystem:
	Missing key(s) in state_dict: "evaluator.lpips.scaling_layer.shift", "evaluator.lpips.scaling_layer.scale", "evaluator.lpips.net.slice1.0.weight", "evaluator.lpips.net.slice1.0.bias", "evaluator.lpips.net.slice2.3.weight", "evaluator.lpips.net.slice2.3.bias", "evaluator.lpips.net.slice3.6.weight", "evaluator.lpips.net.slice3.6.bias", "evaluator.lpips.net.slice4.8.weight", "evaluator.lpips.net.slice4.8.bias", "evaluator.lpips.net.slice5.10.weight", "evaluator.lpips.net.slice5.10.bias", "evaluator.lpips.lin0.model.1.weight", "evaluator.lpips.lin1.model.1.weight", "evaluator.lpips.lin2.model.1.weight", "evaluator.lpips.lin3.model.1.weight", "evaluator.lpips.lin4.model.1.weight", "evaluator.lpips.lins.0.model.1.weight", "evaluator.lpips.lins.1.model.1.weight", "evaluator.lpips.lins.2.model.1.weight", "evaluator.lpips.lins.3.model.1.weight", "evaluator.lpips.lins.4.model.1.weight".

I'm not sure but maybe you did intend to load only a certain part of class AnimeNeRFSystem (i.e. self.models), however the pytorch-lighting load_from_checkpoint fucntion couldn't load appropriately the weights and bias from checkpoint file.

I just changed the strict=True parameter of load_from_checkpoint function to load pre-trained model, and now the question:

Is this a right direction for executing code?

If right, could you change your source code?