lizhe00 / animatablegaussians Goto Github PK

Hello, is there a date expected to release the code?

How do I get smpl_params in my own dataset？

Could you share some method/repo that predict the smpl mesh from the calibrated multi-view cameras?

Hi, thank you for the amazing work!

I just have a question regarding the inputs of the StyleUNet.

According to the paper, your StyleUNets takes as inputs both front and back posed position maps and outputs the front and back pose-dependent gaussian maps.

Meanwhile, I noticed that only the front posed position map is used as the condition to predict both front and back gaussian maps.

I wonder whether this is intentionally done since the outputs still look good.

Thank you in advance.

Great work on the new relightable extension.
As claimed in the paper, that Cam127 was used for the evaluation in Table IV.

The numerical results are computed on the 48-548 frames and the “Cam127” camera view in the “Actor01/Sequence1” from ActorsHQ dataset.

Questions are:

1. Was the Cam127 used for training?
2. Was the training conducted on the whole sequence (including 48-548)?
3. Was the setting the same as the avatar.yaml files in this repo? e.g. the used_cam_ids

https://scikit-learn.org/stable/model_persistence.html#security-maintainability-limitations
warnings.warn(
Traceback (most recent call last):
File "/nas/project/AnimatableGaussians/main_avatar.py", line 837, in
trainer.test()
File "/home/gao/anaconda3/envs/AniGaussian/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/nas/project/AnimatableGaussians/main_avatar.py", line 535, in test
testing_dataset = PoseDataset(**self.opt['test']['pose_data'], smpl_shape = training_dataset.smpl_data['betas'][0])
File "/home/gao/anaconda3/envs/AniGaussian/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/nas/project/AnimatableGaussians/dataset/dataset_pose.py", line 59, in init
raise ValueError('Invalid data_path!')
ValueError: Invalid data_path!

Great work!! I want to know where the code for diffusing the weights from the SMPL surface throughout the whole 3D volume is. Thanks for your help

i want to try on another dataset

what about face movement

Thanks for the inspiring work!

I would like to know by which tool you generate calibration_full.json for each avatar?

Is the tool open-sourced?

Thx in advance!

Getting this error for all files related to avatarrex_zzr

Hello, is there a date expected to release the code?

This work looks amazing!!!

Hi. Thank you for releasing the code of the awesome work!

I'm staying tuned to the preprocessed dataset of ActorsHQ and THuman4.0.

When do you plan to release them?

Best Regards,

I followed the process in gen_weight_volume.py to generate cano_weight_volume, and I referenced the code at

如果我想创建自己的骨架动作，即生成自己的.npz文件，但我在THUman4.0的主页中没有看到这一部分的介绍，是还没有开源吗

When I tried your method of processing camera parameters in the actorshq dataset and applied the processed extrinsic and intrinsic matrices to the original 3DGS, the results were very blurry. Have you tried the same method before？

Hi, thanks for your nice work. I have read your paper, but I did not find any information about the viewpoint settings during training and testing. I noticed that all 16 viewpoints were used during training in your code, but can you clarify which viewpoints were used during testing? For example, which viewpoints were the results in Table 2 of the paper based on? Could you provide some instructions regarding the frame and camera viewpoint configuration during testing?

In line 52 of gaussian_renderer.py, camera_center = torch.linalg.inv(extr)[:3, 3] seems to be changed to camera_center = torch.linalg.inv(extr)[3, :3], although judging from the results It seems to have little impact.

It looks extremely similar.

It is really a fantastic work. Congrats!
I managed to deal with all the data preprocessing and testing.
However I found the rendered results on avatar_lbn1 is not quite ideal, even on the training smpl dataset.
The rendered image rendered below ith command [python main_avatar.py -c configs/avatarrex_lbn1/avatar.yaml --mode=test]:


Is that sounds reasonable to you?
Thanks very much!

Hello,

First of all, thank you for your excellent work on this project. I have been working with the avatar.py file and came across a line of code that I can't understand: 'pose_map = items['smpl_pos_map'][:3]' Could you please explain why [:3] is used here instead of just using the entire smpl_pos_map like this? 'pose_map = items['smpl_pos_map']' . My understanding is that pose_map should include both the front and back of the human body. Does [:3] serve a specific purpose that I'm missing?

Thank you for your time and assistance.

您好，文档中只介绍了main_avatar.py的用法，我感觉是利用多视角照片和相机位姿训练得到人体三维信息，然后利用已有的骨架动作驱动，这样的话只能对其进行定性评估，因为定量评估没有gt图片。

不知道这个main_template.py是不是类似传统的多视角重建，以gt图片的动作作为模板生成图片，这样就可以进行定量评估了。但是这个main_template.py只有训练模式，得到net.pt，之后该如何操作呢

Hi,

Thank you for your amazing work!

I noticed that you recently updated your website/github for relightable models. Any plan to release either the code or a technical report for the relightable models?

best

When I ran the model test for lbn1 on the terminal, it pointed out that calibration_full.json does not exist in my dataset path. I checked that the lbn1 file in the dataset does not exist, and where should I find it

你好，如果我想将这些视频帧处理成512x512分辨率，应该怎么做呢？这是否涉及到相机参数或者smplx参数的修改？
谢谢！

Hi, thanks for releasing the code! I'm wondering if it's possible to train/test with a batch size bigger than 1? What's the reason a batch size of 1 is used for training? Thanks!

Hello,

I attempted to download the AMASS dataset and use amass_data/CMU/06/06_13_poses.npz to animate the avatar, but I encountered the following error:

Traceback (most recent call last):
File "main_avatar.py", line 845, in
trainer.test()
File "C:\anaconda\envs\test\lib\site-packages\torch\utils_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "main_avatar.py", line 537, in test
testing_dataset = PoseDataset(**self.opt['test']['pose_data'], smpl_shape = training_dataset.smpl_data['betas'][0])
File "C:\anaconda\envs\test\lib\site-packages\torch\utils_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "C:\Users\admin\Desktop\tmr_re\AG\dataset\dataset_pose.py", line 90, in init
smpl_data = {k: torch.from_numpy(v).to(torch.float32) for k, v in smpl_data.items()}
File "C:\Users\admin\Desktop\tmr_re\AG\dataset\dataset_pose.py", line 90, in
smpl_data = {k: torch.from_numpy(v).to(torch.float32) for k, v in smpl_data.items()}
TypeError: can't convert np.ndarray of type numpy.str_. The only supported types are: float64, float32, float16, complex64, complex128, int64, int32, int16, int8, uint8, and bool.

I reviewed the motion sequences you provided and found that the contents of the two are different:
"""
global_orient
transl
body_pose
jaw_pose
betas
expression
left_hand_pose
right_hand_pose
"""
The above is from /thuman4/pose_00.npz.

Here is what I used:
"""
trans
gender
mocap_framerate
betas
dmpls
poses
"""
How should I handle this to make it directly usable as motion input?
I know that the error occurs because the value for gender is a string (str), but how should I handle the other keys?

Hi,I wonder how long does the model take to train? I use one 3090 and have trained about 3 days...

I download your preprocessed files from [PREPROCESSED_DATASET.md].(https://github.com/lizhe00/AnimatableGaussians/blob/master/PREPROCESSED_DATASET.md)
Here is the detailed error:
Import AvatarNet from network.avatar
[ WARN:[email protected]] global /io/opencv/modules/imgcodecs/src/loadsave.cpp (239) findDecoder imread_('media/data4/yejr/AvatarReX/avatarrex_lbn1/smpl_pos_map/cano_smpl_pos_map.exr'): can't open/read file: check file path/integrity
Traceback (most recent call last):
File "main_avatar.py", line 829, in
trainer = AvatarTrainer(config.opt)
File "main_avatar.py", line 48, in init
self.avatar_net = AvatarNet(self.opt['model']).to(config.device)
File "/home/yejr/Digital_Avater/AnimatableGaussians-master/network/avatar.py", line 28, in init
self.cano_smpl_map = torch.from_numpy(cano_smpl_map).to(torch.float32).to(config.device)
TypeError: expected np.ndarray (got NoneType)
How should I do?

May I ask where is the StyleUNet that predicts the intrinsics(normal，Albedo,Roughness&Visibility)?I only found the StyleUNet that predicts the 3Dgaussian properties.

But I can't find the code about PBR and intrinsics.

Thank you for your assistance!

Hi, I want to run comparison_body_only_avatars.py but don't know how to get the needed files, for example ./test_results/subject00/...
Could you provide these files? Thanks a lot!

Error: No matching distribution found for igl==2.2.1

Description

When trying to install the required dependencies from the requirements.txt file, I encountered the following error:

Collecting glfw==2.4.0 (from -r requirements.txt (line 1))
Using cached glfw-2.4.0-py2.py27.py3.py30.py31.py32.py33.py34.py35.py36.py37.py38-none-manylinux2014_x86_64.whl.metadata (4.8 kB)
ERROR: Could not find a version that satisfies the requirement igl==2.2.1 (from versions: none)
ERROR: No matching distribution found for igl==2.2.1

This error occurs when trying to install the `igl` package at version 2.2.1, which appears to be listed in the `requirements.txt` file.

Environments Tested

I have tried the following environments:

1. venv with Python 3.11
2. Anaconda with Python 3.10
3. Anaconda with Python 3.9

Steps to Reproduce

1. Create a new virtual environment (using pyenv or Anaconda)
2. Install the required dependencies from the requirements.txt file

Expected Behavior

All the dependencies listed in the requirements.txt file should be installed successfully without any errors.

Additional Information

• cuda:12.2.0
• ubuntu22.04

Can you please let me know if you have an environment that works well for these errors?
Thank you in advance for your time.

How long the training is？

Thank you for your excellent work and providing the codes.

I want to train on our own dataset. For the setting, how can I obtain smpl_params.npz? Should I use PyMAF-X as mentioned in the AvatarRex? If so, how can I tune the parameters to align it to the calibrated camera parameters (K[R|T])? I was curious about it because smpl parameters obtained from PyMAF-X are trained on fixed virtual camera (maybe focal length=(5000,5000) and extrinsic matrices are identity) across all the images.

I would appreciate it if you could answer.

I use the link https://github.com/lizhe00/AnimatableGaussians/blob/master/PREPROCESSED_DATASET.md to download the preprocessed avatarrex data. However, the error report reminds me that it lacks the smpl_params.npz.

(m3dgs) (base) duantong@user-R8428-A12:/data/duantong/mazipei/AnimatableGaussians$ python main_avatar.py -c configs/avatarrex_zzr/avatar.yaml --mode=train
# Using Pytorch3d Renderer
Import AvatarNet from network.avatar
# Parameter number of AvatarNet is 223648936
Traceback (most recent call last):
  File "main_avatar.py", line 834, in <module>
    trainer.pretrain()
  File "main_avatar.py", line 269, in pretrain
    self.dataset = MvRgbDataset(**self.opt['train']['data'])
  File "/data/duantong/mazipei/AnimatableGaussians/dataset/dataset_mv_rgb.py", line 393, in __init__
    super(MvRgbDatasetAvatarReX, self).__init__(
  File "/home/duantong/anaconda3/envs/m3dgs/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/data/duantong/mazipei/AnimatableGaussians/dataset/dataset_mv_rgb.py", line 40, in __init__
    self.load_smpl_data()
  File "/data/duantong/mazipei/AnimatableGaussians/dataset/dataset_mv_rgb.py", line 253, in load_smpl_data
    smpl_data = np.load(self.data_dir + '/smpl_params.npz', allow_pickle = True)
  File "/home/duantong/anaconda3/envs/m3dgs/lib/python3.8/site-packages/numpy/lib/npyio.py", line 405, in load
    fid = stack.enter_context(open(os_fspath(file), "rb"))
FileNotFoundError: [Errno 2] No such file or directory: './avatar_data/avatarrex/zzr/smpl_params.npz'

In the process of creating Avatar Animations, you may find that certain pose-driven animations deliver superior quality compared to others. For example, the "THuman4.0_POSE" may provide more convincing results than "MPI_mosh". How to improve the quality of driving any arbitrary action?

Hi, thanks again for releasing the code! If I'd like to retrain the model to reduce the VRAM usage, what are some key parameters/configurations I can tune? Thanks!

First, I'd like to express my gratitude for your outstanding work and congratulations on your acceptance to CVPR 2024!

I am currently testing he provided model on the AvatarRex dataset to animate avatars, and it works wonderfully. However, I am facing some confusion regarding the process of reconstructing an avatar from an arbitrary self-shot video.

According to GEN_DATA.md, it seems that I need to provide a dataset for a single avatar. Given that the THuman4.0 dataset uses 24 cameras for each avatar, does this imply that I need to create an independent dataset for each new avatar I wish to reconstruct?

This requirement seems to suggest that the reconstruction cost for each avatar might be high. Could you please guide me on whether I have misunderstood the process?

Thank you for your assistance!

Hello, I met some trouble when run the command "python main_avatar.py -c configs/avatarrex_zzr/avatar.yaml --mode=test".
For I did not find the "prev_ckpt" in the link(the pretrained model, some thing like "net.py"?):
https://github.com/lizhe00/AnimatableGaussians/blob/master/PRETRAINED_MODEL.md

Thank you for your reply.