Which checkpoint is used in Unet module of UNetMV2DConditionModel about gaussianeditor HOT 8 OPEN

@zjcs Hello, I have a similar issue. Did you manage to resolve it?

Please refer the above discussion.

from gaussianeditor.

You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
Some weights of UNetMV2DConditionModel were not initialized from the model checkpoint at lambdalabs/sd-image-variations-diffusers and are newly initialized because the shapes did not match:

• conv_in.weight: found shape torch.Size([320, 4, 3, 3]) in the checkpoint and torch.Size([320, 8, 3, 3]) in the model instantiated
  You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
  use xformers.

When I use cfg.pretrained_model_name_or_path in UNetMV2DConditionModel.from_pretrained_2d, the above log occurs.

The checkpoint is wrong?

from gaussianeditor.

We use wonder3D to implement our add function. It looks like you have problem downloading the Wonder3D ckpt. I have never encountered this problem before. Could you please tell me the way you prepare your environment? Or you can try download the ckpt from wonder3D manually.

from gaussianeditor.

Did you run the sh download wonder3d.sh command?

from gaussianeditor.

from gaussianeditor.

from gaussianeditor.

Did you run the sh download wonder3d.sh command?

Sorry to reply late.

I just directly run the code and the checkpoint download directly via the huggingface.
At previouse time, I did two modifies following the origin wonder3D, and the final result is right:

1. threestudio/utils/wonder3D/configs/mvdiffusion-joint-ortho-6views.yaml

#pretrained_model_name_or_path: 'lambdalabs/sd-image-variations-diffusers'
#pretrained_unet_path: './ckpts/'
pretrained_model_name_or_path: 'flamehaze1115/wonder3d-v1.0'

2. threestudio/utils/wonder3D/test_mvdiffusion_seq.py

#unet = UNetMV2DConditionModel.from_pretrained_2d(cfg.pretrained_unet_path, subfolder="unet", revision=cfg.revision, **cfg.unet_from_pretrained_kwargs)
unet = UNetMV2DConditionModel.from_pretrained_2d(cfg.pretrained_model_name_or_path, subfolder="unet", revision=cfg.revision, **cfg.unet_from_pretrained_kwargs)

I will try bash wonder3d.sh later.

BTW, why these two places is different to the origin Wonder3D code? A better fine-tune checkpoint?

from gaussianeditor.

@zjcs Hello, I have a similar issue. Did you manage to resolve it?

from gaussianeditor.