Bad results than expected. about 4k4d HOT 10 CLOSED

Hi, looks like you loaded and tried to render the whole sequence. May I ask what command did you use?

from 4k4d.

I re-run the scripts:

# Render 1 frame every 10 frames, totally 15 frames
evc -t gui -c configs/projects/realtime4dv/rendering/4k4d_basketball.yaml val_dataloader_cfg.dataset_cfg.frame_sample=0,150,10

# Render first 30 frames
evc -t gui -c configs/projects/realtime4dv/rendering/4k4d_basketball.yaml val_dataloader_cfg.dataset_cfg.frame_sample=0,30,1

on my end and the results looks fine. So I'm suspecting corrupted dataset. Maybe you can follow the guide in the readme to downloaded my processed dataset to try the rendering?

from 4k4d.

ok, now It seems quite well, I changed directly the yaml file, which may cause the bad result.

But when I look upward the basketball player, the result still bad (0:06s in my uploaded video) Is it normal?

from 4k4d.

The artifact at 0:06s is caused by wrongly selected source cameras, which might not cover all the points thus resulting in unreasonable colors. I trained the model with 8 source views but selecting a few more (like 16) should not cause too much issues.
You can specify the number of source views to use for rendering with model_cfg.sampler_cfg.n_srcs=16

from 4k4d.

from 4k4d.

I added "model_cfg.sampler_cfg.n_srcs=16" and it looks good. But what's the meaning of "trained the model with 8 source views but selecting a few more (like 16)"? Does it mean that you use 8 views (sparse view) for training while using 16 views for IBR-like rendering?

from 4k4d.

So originally, if I don't add "model_cfg.sampler_cfg.n_srcs=16", it will use 8 cameras to render? And the images loaded are from 8 cameras?

from 4k4d.

Does it mean that you use 8 views (sparse view) for training while using 16 views for IBR-like rendering?

During training, the pipeline of rendering an image is the same as those during inference. This means we also have to sample some source views for IBR, which is set to 8 in this model.
We used all views for training in this model, which means we render images on all views and compare with GTs to compute the loss. During the process of rendering an image, we can sample 8 views or 16 view for IBR, which is not the same as the number of views used for training.

So originally, if I don't add "model_cfg.sampler_cfg.n_srcs=16", it will use 8 cameras to render? And the images loaded are from 8 cameras?

The sampling process of IBR is a simple closest neighbor algorithm. From all input views (72), we sample 8 source views for IBR. All 72 images for the basketball dataset is loaded, but when rendering a particular image, only the closest 8 views are used.

Hope this clarifies~

from 4k4d.

The key is to differentiate between source views (for IBR) and training views (for optimization).

from 4k4d.

OK, now I see, Thanks again for your kind and prompt reply!

from 4k4d.