Is the training based on LoRA or just tune the original model parameters! about zero123plus HOT 3 CLOSED

And another thing that confuses me is that "scaled by a factor of 5" in the paper, is that implemented in the code? I could not find that.

from zero123plus.

1. Original parameters; LoRA would barely work if the rank is as low as 4, but can work to some extent if the rank is more than 64.
2. ControlNet generally assume local spatial correlations between input condition and output image, so you will need multi-view control images; for off-the-shelf ones you may want to try https://github.com/haofanwang/ControlNet-for-Diffusers; I am not familiar with T2I-Adapter so I cannot tell now. The training process is after the training of UNet, the same as regular ControlNets.
3. It is not possible; in general you will need other tricks such as Gaussian blob initialization (I am referring to the Instant3D paper by Adobe) to provide more global clues if you do not change the schedule. I think changing the schedule is the principly-correct way to go though. Epsilon models can be used to enhance local details, which is one of the future works I mentioned in Zero123++ report and I am currently working on.

from zero123plus.

By default the SD VAE output needs to be rescaled by about 0.18 (vae.config.scaling_factor) before sending into diffusion; we skipped that step for the condition branch (so it is roughly scaled by a factor of 5), and we have an extra function called scale_latents that normalizes the residual by shifting and rescaling the latents according to statistics we compute with Objaverse renders.

from zero123plus.