river-zhang / sifu Goto Github PK

Hi authors, could you provide some details on your evaluation protocol for Table 6? Specifically,

1. what is the rendering size?
2. how many views per subject?
3. which backbone for lpips?

Thank you very much!

Body Fitting --- normal: 0.104 | silhouette: 0.054 | Total: 0.158
d5acc65c50a71c7c54eb38323966f6c1_smpl.obj correct.

ICON infer error.
d5acc65c50a71c7c54eb38323966f6c1_recon.obj view：

Cloth Refinement --- cloth:3.38728 | stiffness:0.66788 | rigid:0.04957 | laplacian:0.28668 | Total: 4.39141

d5acc65c50a71c7c54eb38323966f6c1_cloth.gif view：

thanks greate work!

Hi, thank you for releasing the code. I tried the demo code but got this error.

FileNotFoundError: [Errno 2] No such file or directory: './data/smpl_related/smpl_data/smplx_vertex_lmkid.npy'

Could you please share this file?

Thanks!

hi there, the inference is ok, but then i ran into an error while testing, i am not sure what caused this error and how to fix it
my GPU is a Tesla V100-SXM2-32GB
here is the error info

(sifu) lby@ubuntu:~/code/SIFU$ python -m apps.train -cfg ./configs/train/sifu.yaml -test

ICON:
w/ Global Image Encoder: True
Image Features used by MLP: ['normal_F', 'normal_B']
Geometry Features used by MLP: ['sdf', 'cmap', 'norm', 'vis', 'sample_id']
Dim of Image Features (local): 6
Dim of Geometry Features (ICON): 7
Dim of MLP's first layer: 78

GPU available: True, used: True
TPU available: None, using: 0 TPU cores
Resume MLP weights from ./data/ckpt/sifu.ckpt
Resume normal model from ./data/ckpt/normal.ckpt
load from ./data/cape/test.txt
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
Testing: 0it [00:00, ?it/s]../aten/src/ATen/native/cuda/MultinomialKernel.cu:109: binarySearchForMultinomial: block: [7,0,0], thread: [96,0,0] Assertion `cumdist[size - 1] > static_cast<scalar_t>(0)` failed.
../aten/src/ATen/native/cuda/MultinomialKernel.cu:109: binarySearchForMultinomial: block: [7,0,0], thread: [97,0,0] Assertion `cumdist[size - 1] > static_cast<scalar_t>(0)` failed.
......(omit)
../aten/src/ATen/native/cuda/MultinomialKernel.cu:109: binarySearchForMultinomial: block: [2,0,0], thread: [95,0,0] Assertion `cumdist[size - 1] > static_cast<scalar_t>(0)` failed.
Traceback (most recent call last):
  File "/home/lby/miniconda3/envs/sifu/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/lby/miniconda3/envs/sifu/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/lby/code/SIFU/apps/train.py", line 157, in <module>
    trainer.test(model=model, datamodule=datamodule)
  File "/home/lby/miniconda3/envs/sifu/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 915, in test
    results = self.__test_given_model(model, test_dataloaders)
  File "/home/lby/miniconda3/envs/sifu/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 973, in __test_given_model
    results = self.fit(model)
  File "/home/lby/miniconda3/envs/sifu/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 499, in fit
    self.dispatch()
  File "/home/lby/miniconda3/envs/sifu/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 540, in dispatch
    self.accelerator.start_testing(self)
  File "/home/lby/miniconda3/envs/sifu/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 76, in start_testing
    self.training_type_plugin.start_testing(trainer)
  File "/home/lby/miniconda3/envs/sifu/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 118, in start_testing
    self._results = trainer.run_test()
  File "/home/lby/miniconda3/envs/sifu/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 786, in run_test
    eval_loop_results, _ = self.run_evaluation()
  File "/home/lby/miniconda3/envs/sifu/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 725, in run_evaluation
    output = self.evaluation_loop.evaluation_step(batch, batch_idx, dataloader_idx)
  File "/home/lby/miniconda3/envs/sifu/lib/python3.8/site-packages/pytorch_lightning/trainer/evaluation_loop.py", line 160, in evaluation_step
    output = self.trainer.accelerator.test_step(args)
  File "/home/lby/miniconda3/envs/sifu/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 195, in test_step
    return self.training_type_plugin.test_step(*args)
  File "/home/lby/miniconda3/envs/sifu/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 134, in test_step
    return self.lightning_module.test_step(*args, **kwargs)
  File "/home/lby/code/SIFU/apps/ICON.py", line 686, in test_step
    chamfer, p2s = self.evaluator.calculate_chamfer_p2s(num_samples=1000)
  File "/home/lby/code/SIFU/lib/dataset/Evaluator.py", line 167, in calculate_chamfer_p2s
    sample_points_from_meshes(self.tgt_mesh, num_samples))
  File "/home/lby/code/pytorch3d/pytorch3d/ops/sample_points_from_meshes.py", line 100, in sample_points_from_meshes
    sample_face_idxs += mesh_to_face[meshes.valid].view(num_valid_meshes, 1)
RuntimeError: numel: integer multiplication overflow
Testing:   0%|          | 0/450 [00:06<?, ?it/s]

https://www.dropbox.com/s/aeulffqzb3zmh8x/pare-github-data.zip

"Is it the case that the code specifies using GPU number 0, and if I want to use GPU number 1, does the command 'CUDA_VISIBLE_DEVICES=1 python -m apps.infer -cfg ./configs/sifu.yaml -gpu 0 -in_dir ./examples -out_dir ./results -loop_smpl 100 -loop_cloth 200 -hps_type pixie' have no effect, and manual modification of the code is required?"

Hello, could you please share the modified PIFu's rendering code for rendering the THUman2.0 dataset's UV files? The original code was for rendering RenderPeople's dataset.

GOOD JOB. several questions about the code.

1. The Values of default "mcube_res" is incosistent in two codes.
   https://github.com/River-Zhang/SIFU/blob/ca76e36e929da7dd0f0d7e617bee19867bcfb046/README.md?plain=1#L87
   https://github.com/River-Zhang/SIFU/blob/ca76e36e929da7dd0f0d7e617bee19867bcfb046/configs/train/sifu.yaml#L68

2. I noticed that you modified the rendering script for the dataset. Based on your script, I re-rendered the dataset. I think you modified the script mainly to obtain side-view data of the human body. I want to know if the texture training data was rendered using the script from ICON or from the PIFU training data.

'uv_render_path': osp.join(self.root, render_folder, 'uv_color', f'{rotation:03d}.png'),
'uv_mask_path': uv_mask_path,
'uv_pos_path': uv_pos_path,
'uv_normal_path': osp.join(self.root, render_folder, 'uv_normal', '%02d.png' % (0)),
'image_mask_path':image_mask_path,
'param_path':param_path,

3.Are the geometric and texture stages trained together?
error=BCEloss+color_loss

4. Will there be a detailed document about the training, as well as the the code of texure reconstruction, including test and inference?

sifu.ckpt load success
normal.ckpt load failed
EOFError: Ran out of input

Traceback (most recent call last):
File "/home/j222/anaconda/envs/sifu/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/home/j222/anaconda/envs/sifu/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/home/j222/SIFU-main/apps/train.py", line 18, in
from lib.dataset.PIFuDataModule import PIFuDataModule
File "/home/j222/SIFU-main/lib/dataset/PIFuDataModule.py", line 3, in
from .PIFuDataset import PIFuDataset
File "/home/j222/SIFU-main/lib/dataset/PIFuDataset.py", line 31, in
import vedo
File "/home/j222/anaconda/envs/sifu/lib/python3.8/site-packages/vedo/init.py", line 31, in
from vedo.plotter import *
File "/home/j222/anaconda/envs/sifu/lib/python3.8/site-packages/vedo/plotter.py", line 4, in
import vtk
File "/home/j222/anaconda/envs/sifu/lib/python3.8/site-packages/vtk.py", line 47, in
from vtkmodules.vtkIOXdmf2 import *
ImportError: /home/j222/anaconda/envs/sifu/lib/python3.8/site-packages/vtkmodules/../../.././././libcurl.so.4: undefined symbol: nghttp2_option_set_no_rfc9113_leading_and_trailing_ws_validation

Thanks. The test results of the pre-trained model are exactly the same as yours. However, the results of my own training are very poor, and I don't know why.

The test result is as follow.

Hello, I would like to ask if it is not necessary to have the left and right normal maps of the SMPL model when testing with the CAPE dataset? I noticed that the dataset does not include them.

Glad to see this project!Can cuda=11.4 environment deploy this project?

Good afternoon!

I tried to set up the environment on a clean Ubuntu 20.04 (wsl Windows 11), following the instructions:

git clone https://github.com/River-Zhang/SIFU.git
sudo apt-get install libeigen3-dev ffmpeg
cd SIFU
conda env create -f environment.yaml
conda activate sifu
pip install -r requirements.txt

However, I encountered several missing dependencies, conflicts, and other issues. For example, I needed to first install build essentials with:
sudo apt-get install build-essential

And for the mlt package, it is necessary to install version 2024.0.0, otherwise, an error occurs pytorch/pytorch#123097.

Eventually, I forked the repository https://github.com/levnikolaevich/SIFU and tried to fix the packages, but I gave up when I encountered an error trying to execute the command:

python -m apps.infer -cfg ./configs/sifu.yaml -gpu 0 -in_dir ./examples -out_dir ./results -loop_smpl 100 -loop_cloth 200 -hps_type pixie

The error message was:

Traceback (most recent call last):
  File "/home/lev/miniconda3/envs/sifu/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/lev/miniconda3/envs/sifu/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/lev/SIFU/apps/infer.py", line 30, in <module>
    from lib.common.render import query_color, image2vid
  File "/home/lev/SIFU/lib/common/render.py", line 17, in <module>
    from pytorch3d.renderer import (
  File "/home/lev/miniconda3/envs/sifu/lib/python3.8/site-packages/pytorch3d/renderer/__init__.py", line 3, in <module>
    from .blending import (
  File "/home/lev/miniconda3/envs/sifu/lib/python3.8/site-packages/pytorch3d/renderer/blending.py", line 9, in <module>
    from pytorch3d import _C
ImportError: libcudart.so.10.1: cannot open shared object file: No such file or directory

=============

Could you please commit an environment.yml file created with the command in an environment where everything works for you?

conda env export > environment.yml

Then, you can remove the requirements.txt file, which currently lists the package pymeshlab twice.

Thank you in advance, and congratulations on your acceptance to CVPR 2024!

Fantastic job! Could you please tell me when will you release the code?

Hello, I have successfully tested the pre-trained model, but I found that the results do not match the data in the benchmark, especially for CAPE-NFP. Is this normal? Looking forward to your reply.

Which file in SIFU can I get the rendered 3D human model? I would like to see a picture of the back of the figure as in the paper.

How to display the color of a 3D human model for inference?

Excuse me, the checkpoint cannot be downloaded because of the webpage error. Please check it out if possible. Thank you!