apchenstu / mvsnerf Goto Github PK

[ICCV 2021] Our work presents a novel neural rendering approach that can efficiently reconstruct geometric and neural radiance fields for view synthesis.

License: MIT License

Python 53.24% Jupyter Notebook 46.76%

nerf 3d-reconstruction

mvsnerf's People

Contributors

Stargazers

Watchers

Forkers

dingyikang wx-b wps1215 kwea123 plsang chuong yannnnnnnnnnnn thisisisaac zcong17huang shanghaitech-vrvc-center findeton ooxpyctoo zengxianxian727 zhangmozhe drealarun ranqing msfrms peterzhousz cv-ip dongwoohhh ohhandsome stephanie-ustc freemang zishanqin tarataro yshen47 shrubb ck624 fengqiliu1221 zhaofuq corneliushsiao cduguet derrick-xwp hansongfang gitshohoku gabrielscheibler cdq14 dmarx hengfei-wang xuyuewei anonymous20221108 yuhangming laphisboy felizang hulaba bruinxiong mysterycver freeairka yanjiang0216 iuk cmu-learning-3d topworld2040 bj4-567 850207131 davidfjsmith rudyryk aloksmx atlasgooo2 tangrui2018 autohdr whuhxb ymclsh yuchen-song wujinzhong xx-peach lq826311756 genff peterzs philleer gg-big-org jackzhousz zsh2000 jh-001 yzt-cqu qianqian121 rpo130 eviliclufas thj2333 kongdy1999 daebangstn liangrrr goodyenough xifeng205 andrius-k

mvsnerf's Issues

build_rays from each view?

Hi,

I am a bit confused about this line of code:

mvsnerf/utils.py

Line 177 in 6e01111

for i in range(V-1,V):

Is this intended to go through all the views except the reference view? Assuming that V is 3 (views 0, 1 and 2) and reference view is 0, then it should give you view 1 and 2. Therefore it should be range(V-2,V) (or range(1,V) to generalise to any number of views).

Please, correct me if my understanding is wrong.
Thank you!

Multiple GPU training support

Does this model support multi-GPU training ?
If do support and require, how to set in code ?

How to reproduce results?

Is there pretrained model available?

DTU evaluation accuracy?

Hi, thanks for sharing such a great work!

I have a simple question regarding evaluation accuracy on DTU dataset when using the provided pre-trained checkpint.

I got the numbers below that are slightly lower than the ones in the paper. What do I miss? (For LLFF dataset, I got the same evaluation accuracy as the paper using the same pre-trained checkpoint. I used renderer.ipynb to evaluate as suggested in the repo.)

How to evaluate the no per-scene optimazation model?

Well, I want to know how to evaluate the no per-scene optimazation model on the DTU dataset.

Enquiry into hyperparameters for reproducing results

Hi, thanks for your great work.

I am currently trying to reproduce the results posted on the paper and was wondering on what are the exact hyperparameters used such as the epoch number. Is it same as those posted on the run_batch.py script ? (i.e. epoch count = 8 etc)?

Hope to seek your clarification in this

Thank you

L1 loss usage

Hi,

I was looking through your code (thanks for distributing!) and I have a few questions about the implementation with respect the paper.
In the paper you say that L2 rendering loss is the only loss that you use for supervising this model, but in the following line of code you're using L1 to supervise depth? Am I misunderstanding something?

mvsnerf/train_mvs_nerf_pl.py

Line 41 in 6e01111

self.loss = SL1Loss()

Thanks!

pairs.th file

Hi, thank you for sharing the cool work. I am wondering how to generate the pairs.th file in the configs folder.

issues with running

Invalid type provided for checkpoint_callback: Expected bool but received <class 'pytorch_lightning.callbacks.model_checkpoint.ModelCheckpoint'>. Pass callback instances to the callbacks argument in the Trainer constructor instead.
File "/home/comp/donaldtrump/mvsnerf/train_mvs_nerf_pl.py", line 317, in
amp_level='O1')

What is train_mvs_nerf_fusion_finetuning_pl.py used for?

Thanks for your nice work!

What is train_mvs_nerf_fusion_finetuning_pl.py for when there is already a train_mvs_nerf_finetuning_pl.py ?

Some questions about code details and abnormal phenomenon

Hi, it's very kind of you for releasing the code of the paper, and there are some questions about code details and abnormal phenomenon:

As is shown in the first image, I am bit confused of the codes in 140-144 lines of ./data/dtu.py. The reference should be the first element of "view_ids", the in codes in 140-144 lines you have put in the last position. Is something wrong with my understanding?

And again in the codes in 233-234 lines of ./train_mvs_nerf_pl.py, here appears two notions, tgt and ref. So may I ask what exactly src, ref, and tgt stands for?

Is the "with_depth" option means that we have got the depth map of dataset and we should not activate it if we do not get the depth map?

Last qusetion, when I train with my own dataset, nan appears after a few iterations. Have you ever met with the same problem?

Looking forward to your answers. Thanks for your work again.

Fine-tuning the problem

When I try finetune, the following error appears, how can I solve this?

Multiple GPU training support

Does this model support multi-GPU training ?
If do support and require, how to set in code ?

About the img_downscale

Hi, I have pull down your newest code.
However, I cannot run the finetune script for you have not define the img_scale in train_mvs_nerf_finetuning_pl.py

About Question with depth

Reference to your readme, while the command CUDA_VISIBLE_DEVICES=0 python train_mvs_nerf_pl.py --with_depth --imgScale_test 1.0 \ --expname mvs-nerf-is-all-your-need \ --num_epochs 6 --N_samples 128 --use_viewdirs --batch_size 1024 \ --dataset_name dtu \ --datadir path/to/dtu/data \ --N_vis 6 cannot work.

Error information :

log['val_psnr'] = mse2psnr(torch.mean(img_err_abs[:,mask] ** 2))
IndexError: The shape of the mask [1, 1] at index 0 does not match the shape of the indexed tensor [3, 512, 640] at index1

Missing dependency definition - neither a requirements.txt nor a environment.yml

Hi there,

May I kindly ask you to add a requirements.txt or environment.yml (with the later being prefered)? I am trying to get your code working and am running into multiple dependency issues, such as:

module not found errors for at least three packages, e.g., kornia
ImportError for inplace_abn, which might indicate an incompatible PyTorch version (ImportError: /mvsnerf/conda-env-py37/lib/python3.7/site-packages/inplace_abn/_backend.cpython-37m-x86_64-linux-gnu.so: undefined symbol: _ZNK2at6Tensor3addEN3c106ScalarES2_)

Highly appreciated if you run something like conda env export --file environment.yml from within the active conda environment and add the file to the repository.

Thanks a lot!

finetuning results are much worse than general results, why??

There are other people reporting the same problem on DTU datasets and LLFF datasets. I have the same problem on my customized LLFF dataset. Can author give an answer about this problem?

What's the fps during inference?

Bug with neural scene rendering after fine-tunning

Hello, thank you for releasing the code for this great research project! I ran the code and did the fine-tunning for the LLFF scenes and it seems that the results now seem worse than the first time I ran the code.

I have seen that there were some modifications in the LLFF loader and this might be the cause of this bug, however it seems to me pretty strange. I would be happy if someone could help :)

NeRF 9.5h results

Which implementation do you use to generate these results? They seem much worse than the results reported in the paper and the results I got with my implementation.

Here's my result with horns, PSNR reaches already 31.6 after 9.5h (reaches 25.91 after 22mins btw). I believe there should be some simplification in the implementation that you adopt, because I implement almost the same way as the original paper does, and adopt the same hyperparameters.

code release time!

你好，想问下具体代码开源时间

NaN raw output of network_query_fn()

I've been having many times this error, which appears randomly while finetuning:

[W python_anomaly_mode.cpp:104] Warning: Error detected in PowBackward0. Traceback of forward call that caused the error:
  File "train_mvs_nerf_finetuning_pl.py", line 324, in <module>
    trainer.fit(system)
  File "/root/miniconda3/envs/mvsnerf2/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 458, in fit
    self._run(model)
  File "/root/miniconda3/envs/mvsnerf2/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 756, in _run
    self.dispatch()
  File "/root/miniconda3/envs/mvsnerf2/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 797, in dispatch
    self.accelerator.start_training(self)
  File "/root/miniconda3/envs/mvsnerf2/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 96, in start_training
    self.training_type_plugin.start_training(trainer)
  File "/root/miniconda3/envs/mvsnerf2/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 144, in start_training
    self._results = trainer.run_stage()
  File "/root/miniconda3/envs/mvsnerf2/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 807, in run_stage
    return self.run_train()
  File "/root/miniconda3/envs/mvsnerf2/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 869, in run_train
    self.train_loop.run_training_epoch()
  File "/root/miniconda3/envs/mvsnerf2/lib/python3.8/site-packages/pytorch_lightning/trainer/training_loop.py", line 499, in run_training_epoch
    batch_output = self.run_training_batch(batch, batch_idx, dataloader_idx)
  File "/root/miniconda3/envs/mvsnerf2/lib/python3.8/site-packages/pytorch_lightning/trainer/training_loop.py", line 738, in run_training_batch
    self.optimizer_step(optimizer, opt_idx, batch_idx, train_step_and_backward_closure)
  File "/root/miniconda3/envs/mvsnerf2/lib/python3.8/site-packages/pytorch_lightning/trainer/training_loop.py", line 434, in optimizer_step
    model_ref.optimizer_step(
  File "/root/miniconda3/envs/mvsnerf2/lib/python3.8/site-packages/pytorch_lightning/core/lightning.py", line 1403, in optimizer_step
    optimizer.step(closure=optimizer_closure)
  File "/root/miniconda3/envs/mvsnerf2/lib/python3.8/site-packages/pytorch_lightning/core/optimizer.py", line 214, in step
    self.__optimizer_step(*args, closure=closure, profiler_name=profiler_name, **kwargs)
  File "/root/miniconda3/envs/mvsnerf2/lib/python3.8/site-packages/pytorch_lightning/core/optimizer.py", line 134, in __optimizer_step
    trainer.accelerator.optimizer_step(optimizer, self._optimizer_idx, lambda_closure=closure, **kwargs)
  File "/root/miniconda3/envs/mvsnerf2/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 329, in optimizer_step
    self.run_optimizer_step(optimizer, opt_idx, lambda_closure, **kwargs)
  File "/root/miniconda3/envs/mvsnerf2/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 336, in run_optimizer_step
    self.training_type_plugin.optimizer_step(optimizer, lambda_closure=lambda_closure, **kwargs)
  File "/root/miniconda3/envs/mvsnerf2/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 193, in optimizer_step
    optimizer.step(closure=lambda_closure, **kwargs)
  File "/root/miniconda3/envs/mvsnerf2/lib/python3.8/site-packages/torch/optim/lr_scheduler.py", line 65, in wrapper
    return wrapped(*args, **kwargs)
  File "/root/miniconda3/envs/mvsnerf2/lib/python3.8/site-packages/torch/optim/optimizer.py", line 89, in wrapper
    return func(*args, **kwargs)
  File "/root/miniconda3/envs/mvsnerf2/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/root/miniconda3/envs/mvsnerf2/lib/python3.8/site-packages/torch/optim/adam.py", line 66, in step
    loss = closure()
File "/root/miniconda3/envs/mvsnerf2/lib/python3.8/site-packages/pytorch_lightning/trainer/training_loop.py", line 732, in train_step_and_backward_closure
    result = self.training_step_and_backward(
  File "/root/miniconda3/envs/mvsnerf2/lib/python3.8/site-packages/pytorch_lightning/trainer/training_loop.py", line 823, in training_step_and_backward
    result = self.training_step(split_batch, batch_idx, opt_idx, hiddens)
  File "/root/miniconda3/envs/mvsnerf2/lib/python3.8/site-packages/pytorch_lightning/trainer/training_loop.py", line 290, in training_step
    training_step_output = self.trainer.accelerator.training_step(args)
  File "/root/miniconda3/envs/mvsnerf2/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 204, in training_step
    return self.training_type_plugin.training_step(*args)
  File "/root/miniconda3/envs/mvsnerf2/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 155, in training_step
    return self.lightning_module.training_step(*args, **kwargs)
  File "train_mvs_nerf_finetuning_pl.py", line 177, in training_step
    img_loss = img2mse(rgbs, rgbs_target)
  File "/host/home/ubuntu/mvsnerf/utils.py", line 12, in <lambda>
    img2mse = lambda x, y : torch.mean((x - y) ** 2)
 (function _print_stack)
Traceback (most recent call last):
  File "train_mvs_nerf_finetuning_pl.py", line 324, in <module>
    trainer.fit(system)
  File "/root/miniconda3/envs/mvsnerf2/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 458, in fit
    self._run(model)
  File "/root/miniconda3/envs/mvsnerf2/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 756, in _run
    self.dispatch()
  File "/root/miniconda3/envs/mvsnerf2/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 797, in dispatch
    self.accelerator.start_training(self)
  File "/root/miniconda3/envs/mvsnerf2/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 96, in start_training
    self.training_type_plugin.start_training(trainer)
  File "/root/miniconda3/envs/mvsnerf2/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 144, in start_training
    self._results = trainer.run_stage()
  File "/root/miniconda3/envs/mvsnerf2/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 807, in run_stage
    return self.run_train()
  File "/root/miniconda3/envs/mvsnerf2/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 869, in run_train
    self.train_loop.run_training_epoch()
  File "/root/miniconda3/envs/mvsnerf2/lib/python3.8/site-packages/pytorch_lightning/trainer/training_loop.py", line 499, in run_training_epoch
    batch_output = self.run_training_batch(batch, batch_idx, dataloader_idx)
  File "/root/miniconda3/envs/mvsnerf2/lib/python3.8/site-packages/pytorch_lightning/trainer/training_loop.py", line 738, in run_training_batch
    self.optimizer_step(optimizer, opt_idx, batch_idx, train_step_and_backward_closure)
  File "/root/miniconda3/envs/mvsnerf2/lib/python3.8/site-packages/pytorch_lightning/trainer/training_loop.py", line 434, in optimizer_step
    model_ref.optimizer_step(
  File "/root/miniconda3/envs/mvsnerf2/lib/python3.8/site-packages/pytorch_lightning/core/lightning.py", line 1403, in optimizer_step
    optimizer.step(closure=optimizer_closure)
File "/root/miniconda3/envs/mvsnerf2/lib/python3.8/site-packages/pytorch_lightning/core/optimizer.py", line 214, in step
    self.__optimizer_step(*args, closure=closure, profiler_name=profiler_name, **kwargs)
  File "/root/miniconda3/envs/mvsnerf2/lib/python3.8/site-packages/pytorch_lightning/core/optimizer.py", line 134, in __optimizer_step
    trainer.accelerator.optimizer_step(optimizer, self._optimizer_idx, lambda_closure=closure, **kwargs)
  File "/root/miniconda3/envs/mvsnerf2/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 329, in optimizer_step
    self.run_optimizer_step(optimizer, opt_idx, lambda_closure, **kwargs)
  File "/root/miniconda3/envs/mvsnerf2/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 336, in run_optimizer_step
    self.training_type_plugin.optimizer_step(optimizer, lambda_closure=lambda_closure, **kwargs)
  File "/root/miniconda3/envs/mvsnerf2/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 193, in optimizer_step
    optimizer.step(closure=lambda_closure, **kwargs)
  File "/root/miniconda3/envs/mvsnerf2/lib/python3.8/site-packages/torch/optim/lr_scheduler.py", line 65, in wrapper
    return wrapped(*args, **kwargs)
  File "/root/miniconda3/envs/mvsnerf2/lib/python3.8/site-packages/torch/optim/optimizer.py", line 89, in wrapper
    return func(*args, **kwargs)
  File "/root/miniconda3/envs/mvsnerf2/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/root/miniconda3/envs/mvsnerf2/lib/python3.8/site-packages/torch/optim/adam.py", line 66, in step
    loss = closure()
  File "/root/miniconda3/envs/mvsnerf2/lib/python3.8/site-packages/pytorch_lightning/trainer/training_loop.py", line 732, in train_step_and_backward_closure
    result = self.training_step_and_backward(
  File "/root/miniconda3/envs/mvsnerf2/lib/python3.8/site-packages/pytorch_lightning/trainer/training_loop.py", line 836, in training_step_and_backward
    self.backward(result, optimizer, opt_idx)
  File "/root/miniconda3/envs/mvsnerf2/lib/python3.8/site-packages/pytorch_lightning/trainer/training_loop.py", line 869, in backward
    result.closure_loss = self.trainer.accelerator.backward(
  File "/root/miniconda3/envs/mvsnerf2/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 308, in backward
    output = self.precision_plugin.backward(
  File "/root/miniconda3/envs/mvsnerf2/lib/python3.8/site-packages/pytorch_lightning/plugins/precision/precision_plugin.py", line 79, in backward
    model.backward(closure_loss, optimizer, opt_idx)
  File "/root/miniconda3/envs/mvsnerf2/lib/python3.8/site-packages/pytorch_lightning/core/lightning.py", line 1275, in backward
    loss.backward(*args, **kwargs)
  File "/root/miniconda3/envs/mvsnerf2/lib/python3.8/site-packages/torch/tensor.py", line 245, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "/root/miniconda3/envs/mvsnerf2/lib/python3.8/site-packages/torch/autograd/__init__.py", line 145, in backward
    Variable._execution_engine.run_backward(
RuntimeError: Function 'PowBackward0' returned nan values in its 0th output.

I've been debugging, and so far what I've found is that this is due to this raw output to be NaN in renderer.py:156:

raw = network_query_fn(rays_ndc, angle, input_feat, network_fn)

I'm attaching some scope variables in a pickle to reproduce.

https://www.dropbox.com/s/ew0uetjpedn2e9n/bug_dump.pkl?dl=0

how to better avoid, and mitigate these raw outputs?

the link of the paper is wrong, please check.

"This repository contains a pytorch lightning implementation for the ICCV 2021 paper: MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereo. "

Enquiry into dtu_pairs.txt file

Hi, thanks for your great work.
I would like to make an enquiry into how dtu_pairs.txt file was created (not pairs.th file) and especially into what the floating point values mean in each line.

Thank you

Intrinsic preprocessing

Hello, I have been using your dataloader but I've seen some weird preprocessing you have done. As I recall the training image size is (512,640). Why would you multiply fx, fy, cx, cy by 4 times in this line of code

mvsnerf/data/dtu.py

Line 83 in 6e01111

intrinsic[:2] *= 4

Moreover, later on, you divide the intrinsic again to 4 to produce the projection matrix of each view. Then why would you multiply the intrinsic with 4 in the beginning ?

output NaN when fine-tuning on my own dataset

Hi, thanks for releasing your work! I have some problems when fine-tuning on LLFF and my own dataset.

I wonder why you set spheric_poses=True for LLFF? Besides, when I change it to False(which means to use ndc coordinates), the train_mvs_nerf_finetuning_pl.py gives a poor initial rendering(shown in fig.), and reports error below immediately. It seems that the networks output NaN when training, and show no enhancement after clipping the gradients(to avoid gradient explosion). So do you have any idea of solving this problem? Could there be a numerical error in the code?

I try to fine-tuning on my own dataset, which is sparsely sampled from a real scene dataset and has more complex trajectory than LLFF. It reports the same error as below at the early fine-tuning stage. If it's not caused by numerical error, does it mean that your method is unsuitable for complex posed images of real scene? However, in my comprehension, such scenes should be trained as long time as Nerf in this situation, rather than report NaN when training, right?
Do you have any advice on how to choose source views？For example, should it be very close neighbors or uniformly distributed around the scene? How much co-visibility between source views is proper for your method?

[W python_anomaly_mode.cpp:104] Warning: Error detected in PowBackward0. Traceback of forward call that caused the error:
  File "train_mvs_nerf_finetuning_pl.py", line 309, in <module>
    trainer.fit(system)
  File "/home/wuxin/anaconda3/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 460, in fit
    self._run(model)
  File "/home/wuxin/anaconda3/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 758, in _run
    self.dispatch()
  File "/home/wuxin/anaconda3/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 799, in dispatch
    self.accelerator.start_training(self)
  File "/home/wuxin/anaconda3/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 96, in start_training
    self.training_type_plugin.start_training(trainer)
  File "/home/wuxin/anaconda3/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 144, in start_training
    self._results = trainer.run_stage()
  File "/home/wuxin/anaconda3/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 809, in run_stage
    return self.run_train()
  File "/home/wuxin/anaconda3/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 871, in run_train
    self.train_loop.run_training_epoch()
  File "/home/wuxin/anaconda3/lib/python3.8/site-packages/pytorch_lightning/trainer/training_loop.py", line 499, in run_training_epoch
    batch_output = self.run_training_batch(batch, batch_idx, dataloader_idx)
  File "/home/wuxin/anaconda3/lib/python3.8/site-packages/pytorch_lightning/trainer/training_loop.py", line 738, in run_training_batch
    self.optimizer_step(optimizer, opt_idx, batch_idx, train_step_and_backward_closure)
  File "/home/wuxin/anaconda3/lib/python3.8/site-packages/pytorch_lightning/trainer/training_loop.py", line 434, in optimizer_step
    model_ref.optimizer_step(
  File "/home/wuxin/anaconda3/lib/python3.8/site-packages/pytorch_lightning/core/lightning.py", line 1403, in optimizer_step
    optimizer.step(closure=optimizer_closure)
  File "/home/wuxin/anaconda3/lib/python3.8/site-packages/pytorch_lightning/core/optimizer.py", line 214, in step
    self.__optimizer_step(*args, closure=closure, profiler_name=profiler_name, **kwargs)
  File "/home/wuxin/anaconda3/lib/python3.8/site-packages/pytorch_lightning/core/optimizer.py", line 134, in __optimizer_step
    trainer.accelerator.optimizer_step(optimizer, self._optimizer_idx, lambda_closure=closure, **kwargs)
  File "/home/wuxin/anaconda3/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 329, in optimizer_step
    self.run_optimizer_step(optimizer, opt_idx, lambda_closure, **kwargs)
  File "/home/wuxin/anaconda3/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 336, in run_optimizer_step
    self.training_type_plugin.optimizer_step(optimizer, lambda_closure=lambda_closure, **kwargs)
  File "/home/wuxin/anaconda3/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 193, in optimizer_step
    optimizer.step(closure=lambda_closure, **kwargs)
  File "/home/wuxin/anaconda3/lib/python3.8/site-packages/torch/optim/lr_scheduler.py", line 67, in wrapper
    return wrapped(*args, **kwargs)
  File "/home/wuxin/anaconda3/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 26, in decorate_context
    return func(*args, **kwargs)
  File "/home/wuxin/anaconda3/lib/python3.8/site-packages/torch/optim/adam.py", line 66, in step
    loss = closure()
  File "/home/wuxin/anaconda3/lib/python3.8/site-packages/pytorch_lightning/trainer/training_loop.py", line 732, in train_step_and_backward_closure
    result = self.training_step_and_backward(
  File "/home/wuxin/anaconda3/lib/python3.8/site-packages/pytorch_lightning/trainer/training_loop.py", line 823, in training_step_and_backward
    result = self.training_step(split_batch, batch_idx, opt_idx, hiddens)
  File "/home/wuxin/anaconda3/lib/python3.8/site-packages/pytorch_lightning/trainer/training_loop.py", line 290, in training_step
    training_step_output = self.trainer.accelerator.training_step(args)
  File "/home/wuxin/anaconda3/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 204, in training_step
    return self.training_type_plugin.training_step(*args)
  File "/home/wuxin/anaconda3/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 155, in training_step
    return self.lightning_module.training_step(*args, **kwargs)
  File "train_mvs_nerf_finetuning_pl.py", line 165, in training_step
    img_loss = img2mse(rgbs, rgbs_target)
  File "/home/wuxin/OriginDoc/PycharmProjects/mvsnerf-main/utils.py", line 10, in <lambda>
    img2mse = lambda x, y : torch.mean((x - y) ** 2)
 (function _print_stack)
Traceback (most recent call last):
  File "train_mvs_nerf_finetuning_pl.py", line 309, in <module>
    trainer.fit(system)
  File "/home/wuxin/anaconda3/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 460, in fit
    self._run(model)
  File "/home/wuxin/anaconda3/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 758, in _run
    self.dispatch()
  File "/home/wuxin/anaconda3/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 799, in dispatch
    self.accelerator.start_training(self)
  File "/home/wuxin/anaconda3/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 96, in start_training
    self.training_type_plugin.start_training(trainer)
  File "/home/wuxin/anaconda3/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 144, in start_training
    self._results = trainer.run_stage()
  File "/home/wuxin/anaconda3/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 809, in run_stage
    return self.run_train()
  File "/home/wuxin/anaconda3/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 871, in run_train
    self.train_loop.run_training_epoch()
  File "/home/wuxin/anaconda3/lib/python3.8/site-packages/pytorch_lightning/trainer/training_loop.py", line 499, in run_training_epoch
    batch_output = self.run_training_batch(batch, batch_idx, dataloader_idx)
  File "/home/wuxin/anaconda3/lib/python3.8/site-packages/pytorch_lightning/trainer/training_loop.py", line 738, in run_training_batch
    self.optimizer_step(optimizer, opt_idx, batch_idx, train_step_and_backward_closure)
  File "/home/wuxin/anaconda3/lib/python3.8/site-packages/pytorch_lightning/trainer/training_loop.py", line 434, in optimizer_step
    model_ref.optimizer_step(
  File "/home/wuxin/anaconda3/lib/python3.8/site-packages/pytorch_lightning/core/lightning.py", line 1403, in optimizer_step
    optimizer.step(closure=optimizer_closure)
  File "/home/wuxin/anaconda3/lib/python3.8/site-packages/pytorch_lightning/core/optimizer.py", line 214, in step
    self.__optimizer_step(*args, closure=closure, profiler_name=profiler_name, **kwargs)
  File "/home/wuxin/anaconda3/lib/python3.8/site-packages/pytorch_lightning/core/optimizer.py", line 134, in __optimizer_step
    trainer.accelerator.optimizer_step(optimizer, self._optimizer_idx, lambda_closure=closure, **kwargs)
  File "/home/wuxin/anaconda3/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 329, in optimizer_step
    self.run_optimizer_step(optimizer, opt_idx, lambda_closure, **kwargs)
  File "/home/wuxin/anaconda3/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 336, in run_optimizer_step
    self.training_type_plugin.optimizer_step(optimizer, lambda_closure=lambda_closure, **kwargs)
  File "/home/wuxin/anaconda3/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 193, in optimizer_step
    optimizer.step(closure=lambda_closure, **kwargs)
  File "/home/wuxin/anaconda3/lib/python3.8/site-packages/torch/optim/lr_scheduler.py", line 67, in wrapper
    return wrapped(*args, **kwargs)
  File "/home/wuxin/anaconda3/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 26, in decorate_context
    return func(*args, **kwargs)
  File "/home/wuxin/anaconda3/lib/python3.8/site-packages/torch/optim/adam.py", line 66, in step
    loss = closure()
  File "/home/wuxin/anaconda3/lib/python3.8/site-packages/pytorch_lightning/trainer/training_loop.py", line 732, in train_step_and_backward_closure
    result = self.training_step_and_backward(
  File "/home/wuxin/anaconda3/lib/python3.8/site-packages/pytorch_lightning/trainer/training_loop.py", line 836, in training_step_and_backward
    self.backward(result, optimizer, opt_idx)
  File "/home/wuxin/anaconda3/lib/python3.8/site-packages/pytorch_lightning/trainer/training_loop.py", line 869, in backward
    result.closure_loss = self.trainer.accelerator.backward(
  File "/home/wuxin/anaconda3/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 308, in backward
    output = self.precision_plugin.backward(
  File "/home/wuxin/anaconda3/lib/python3.8/site-packages/pytorch_lightning/plugins/precision/precision_plugin.py", line 79, in backward
    model.backward(closure_loss, optimizer, opt_idx)
  File "/home/wuxin/anaconda3/lib/python3.8/site-packages/pytorch_lightning/core/lightning.py", line 1275, in backward
    loss.backward(*args, **kwargs)
  File "/home/wuxin/anaconda3/lib/python3.8/site-packages/torch/tensor.py", line 221, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/wuxin/anaconda3/lib/python3.8/site-packages/torch/autograd/__init__.py", line 130, in backward
    Variable._execution_engine.run_backward(
RuntimeError: Function 'PowBackward0' returned nan values in its 0th output.
Exception ignored in: <function tqdm.__del__ at 0x7fa58a1585e0>
Traceback (most recent call last):
  File "/home/wuxin/anaconda3/lib/python3.8/site-packages/tqdm/std.py", line 1122, in __del__
  File "/home/wuxin/anaconda3/lib/python3.8/site-packages/tqdm/std.py", line 1335, in close
  File "/home/wuxin/anaconda3/lib/python3.8/site-packages/tqdm/std.py", line 1514, in display
  File "/home/wuxin/anaconda3/lib/python3.8/site-packages/tqdm/std.py", line 1125, in __repr__
  File "/home/wuxin/anaconda3/lib/python3.8/site-packages/tqdm/std.py", line 1475, in format_dict
TypeError: cannot unpack non-iterable NoneType object

What is the training parameter to recreate the generalized model in the paper?

What is the training parameter for this model?

About finetune DTU scene

When fine-tuning the DTU scene, I find only 114 is provided the download link, and how to download the other scene dataset

Runtime error

Hi, thanks for releasing your work. I got a weird runtime error when trying to re-train on the Blender dataset. I used the following command:

 python train_mvs_nerf_pl.py --expname mvsnerf_lego_test --num_epochs 6 --use_viewdirs --dataset_name blender --datadir ./data/lego/

Here is the error:

Found ckpts []
GPU available: True, used: True
TPU available: False, using: 0 TPU cores
Traceback (most recent call last):
  File "/localhome/c-morsingher/mvsnerf/train_mvs_nerf_pl.py", line 320, in <module>
    trainer.fit(system)
  File "/localhome/c-morsingher/anaconda3/envs/mvsnerf/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 458, in fit
    self._run(model)
  File "/localhome/c-morsingher/anaconda3/envs/mvsnerf/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 704, in _run
    self.data_connector.prepare_data(model)
  File "/localhome/c-morsingher/anaconda3/envs/mvsnerf/lib/python3.9/site-packages/pytorch_lightning/trainer/connectors/data_connector.py", line 57, in prepare_data
    model.prepare_data()
  File "/localhome/c-morsingher/mvsnerf/train_mvs_nerf_pl.py", line 80, in prepare_data
    self.train_dataset = dataset(root_dir=train_dir, split='train', max_len=-1 , downSample=args.imgScale_train)
TypeError: __init__() got an unexpected keyword argument 'root_dir'

Any idea on what's happening here? Thank you in advance.

RuntimeError: Tried to access nonexistent attribute or method 'call' of type 'torch.kornia.geometry.boxes.Boxes3D'.

torch._C._jit_script_class_compile(qualified_name, ast, defaults, rcb)

RuntimeError:
Tried to access nonexistent attribute or method 'call' of type 'torch.kornia.geometry.boxes.Boxes3D'. Did you forget to initialize an attribute in init()?:
File "miniconda3/envs/nex/lib/python3.8/site-packages/kornia/geometry/boxes.py", line 601
# Due to some torch.jit.script bug (at least <= 1.9), you need to pass all arguments to init when
# constructing the class from inside of a method.
return cls(hexahedrons, raise_if_not_floating_point=False, mode=mode)
~~~ <--- HERE

cost volume constraint - closer camera view

As you said in #7 ,

we think building volume in the target view may provide a better depth quality but is not an efficient way to do the free-viewpoint rendering

I also feel that the fact that the cost volume must be in some reference view has many limitations. Leave alone view extrapolations, how does your method perform when you move the camera closer to the scene (not zoom in but physically place the camera closer to the scene)?

Currently, the LLFF scenes are all captured roughly at the same distance to the scene, so interpolating at this distance seems great as your video in the readme. However I wonder how it performs in the situation I describe above. With the current cost volume, I can think of two problems:

Although this novel view is still view interpolation (it lies inside the frustums of all reference views), the view to the nearest reference view is very far.
The novel view lies inside the cost volume, a situation that is not seen at all at training time.

To my knowledge, traditional MPI methods cannot handle this kind of situation well either. NeRF performs excellently without problem since it reconstructs the whole scene in 3D without relying on reference views in test time. Following is an example where I move the camera very close to the scene (left is the nearest reference view and right is NeRF synthesized result).

Pardon me for not being able to run your code myself. I didn't find an easy way to run your code with specifying a pose...

Question regarding the architecture

Hi, I really appreciate your work. An impressive idea. I want to ask a question regarding the 3D CNN architecture:

It's written, that CBR3D0 takes T which is cost volume of shape (BATCH_SIZE, CHANNELS (32), N_DEPTH, HEIGHT / 4, WIDTH / 4) and I which are concatenated images, as I guess (BATCH_SIZE, 9, 1, HEIGHT, WIDTH). So, there appears to be a shape inconsistency. Could you, please, clarify this one, if possible.
It also states, that CBT3D2 takes outputs from CTB3D1 and CBR3D2, which is a total of 48 channels instead of 64. Is it a typo?
Thanks in advance for your reply!

Own dataset test.

Hi, thanks for your source code. Could you tell me how can I generate my own dataset？Use colmap or some others？Thank you！

code release time! x2

Is everything going as planned and will the code be released anytime soon? Awesome work!

Why using the training mode of MVSNet during the validation?

mvsnerf/train_mvs_nerf_pl.py

Line 182 in 6e01111

self.MVSNet.train()

There are BNs in MVSNet, which are implemented differently from training.

Rendering after finetuning: index out of bounds

I have finetuned using your indications for a blender lego scene:

CUDA_VISIBLE_DEVICES=0  python train_mvs_nerf_finetuning_pl.py  \
    --dataset_name blender --datadir /path/to/nerf_synthetic/lego \
    --expname lego-ft  --with_rgb_loss  --batch_size 1024  \
    --num_epochs 1 --imgScale_test 1.0 --white_bkgd  --pad 0 \
    --ckpt ./ckpts/mvsnerf-v0.tar --N_vis 1

Then using rendering on renderer-video.ipynb I get to the following error:

Found ckpts ['./runs_fine_tuning/lego-ft/ckpts/latest.tar']
Reloading from ./runs_fine_tuning/lego-ft/ckpts/latest.tar
===> training index: [6, 43, 33, 13, 17, 19, 20, 25, 30, 37, 46, 48, 49, 55, 59, 65]
====> ref idx: [6, 43, 33]
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
/tmp/ipykernel_3803/820391564.py in <module>
     54             w2cs, c2ws = pose_source['w2cs'], pose_source['c2ws']
     55             pair_idx = torch.load('configs/pairs.th')[f'{scene}_train']
---> 56             c2ws_render = get_spiral(c2ws_all[pair_idx], near_far_source, rads_scale = 0.6, N_views=60)# you can enlarge the rads_scale if you want to render larger baseline
     57         else:
     58             # neighboring views with position distance

IndexError: index 43 is out of bounds for axis 0 with size 16

Some more info while debugging:

ipdb>  pair_idx
[6, 43, 33, 13, 17, 19, 20, 25, 30, 37, 46, 48, 49, 55, 59, 65] 
ipdb>  c2ws_all.shape
(16, 4, 4)

Finetuning weights makes worse results

Hi, I try to finetune the model, but I get worse results than the results without finetuning. Could you provide me some advices? Thank you!

About training a genelized model

In the readme I found the line
set batch_size to 1 when you are trining a genelized model, you can enlarge it when fine-tuning.
and since the batch_size is already 1 in the outer loop which loads the image, hould I set the ray size to 1 as well?

And why it's necessary to set batch size to 1?

AttributeError: 'Namespace' object has no attribute 'N_importance'

It shows AttributeError: 'Namespace' object has no attribute 'N_importance' during my training of train_mvs_nerf_pl.py

Resolution of images in training

Hi,

I saw your code resizes the input images to 3:2. Can I use images with different resolution ratio to train the model? Do I need to revise some code?

Thanks

Some questions about the source view

Hi, nice work! I have read your source code and have some questions...

It seems that your pretrained model was trained on 3 source views setting, so if I want to use more source views, I need to retrain the model from scratch. Is it right? Or can I simply modify some codes? Or only retrain some modules?
The training time of your pretrained model? I do not find it in your paper...
About the source view selection, in another issue, you mentioned that the source views should be very close. But in my understanding, the source views are used to initialize the volume, so why not use more sparse views (eg. front+back+left+right)? I think it will cover more seen regions?

Waiting for your answer. Thanks for your work again!

depth is not defined in renderer.ipynb

depth-> depth_h ?

the meaning of unpreprocess function

Instructions for custom data

Hi,

Thanks for this great code. I noted that there are no instructions for running this on custom data (the link is broken). Would you mind explaining what is the process to run this with custom data? Your example data has Camera, Depth and Rectified folders. Are all of these necessary? I couldn't find in your paper any indication on how these are generated.

Any help will be greatly appreciated!

About the accuracy

Hi, I have some questions concerning accuracy in the following:

How to evaluate the accuracy for the 16 DTU? I have changed the max_len from 10 to -1 in val_dataset. Does the modification work right and Does there exist other methods for evaluating?
Except #114 scene in DTU, I cannot implement the accuracy of other scenes (1, 8, 21, 103) as the paper said.
Could you please offer the run commands for pretraining and fine-tuning (DTU) to the official accuracy?
And I am sure the dataset is right because the #114 also comes from it.
By the way, is the newest update(597f5) significant, can the older version (736d9a4) be restored its accuracy?

3D mesh!

Hi, I was wondering about the author's opinion! what would be the best way to obtain the 3D meshes at inference time (not finetuned or finetuned)?

[Question] Is it able to train with images of texturelss objects?

I think it requires local feature extraction step in the early step, so I should provide text rich data, if not, it fails to find a good pose during MVS step. Am I right?
Plus, If I use another new feature extractor(like line feature extractor) and SOTA feature matching network, can you guess it would work better?

issue with running

Invalid type provided for checkpoint_callback: Expected bool but received <class 'pytorch_lightning.callbacks.model_checkpoint.ModelCheckpoint'>. Pass callback instances to the callbacks argument in the Trainer constructor instead.
File "/home/comp/donaldtrump/mvsnerf/train_mvs_nerf_pl.py", line 317, in
amp_level='O1')