fnzhan / neural-gauge-fields Goto Github PK

Fangneng Zhan, Lingjie Liu, Adam Kortylewsk, Christian Theobalt
The International Conference on Learning Representations (ICLR), 2023

• 09/2023: We release an improved UV Mapping and learned TriPlane projection, which allow to learn gauge transformation without any regularization.
• 04/2023: This is an initial release of basic implementation. We are working on the extension of this work, full implementation with cleaned code will be released later.

Install environment:

conda create -n Gauge python=3.8
conda activate Gauge
pip install torch torchvision
pip install tqdm trimesh scikit-image opencv-python configargparse lpips imageio-ffmpeg kornia lpips tensorboard

The 3D coordinate is transformed to 2D manifold (e.g., sphere and square) to index neural fields, where explicit (view-dependent) 2D texture can be obtained for scene editing. Enter UV-Mapping directory to play with it.

Five pre-processed scenes from DTU dataset can be downloaded from Google Drive.
Pre-trained checkpoints on the five scenes can be downloaded from Google Drive. Save the checkpoints in UV-Mapping/.

To render with edited texture (e.g., for DTU scan83), run below command (you can also set the target_texture in dtu_test.sh to your custom texture path):

bash dtu_test.sh 83

To render with original texture, set the target_texture to 'None' and run above command.

Run below commands directly (you can also set data_root in dtu_train.sh to your custom dataset path):

bash dtu_train.sh 83

Instead of applying orthogonal projection from 3D space to Triplane, we directly learn a flexible mapping with neural fields driven by rendering loss. Enter TriPlane directory to play with it.

Please refer to Synthetic-NeRF dataset.

Run below command:

python3 main.py --config configs/lego.txt

Set gauge_start in configs/lego.txt to a number larger than the total iterations, e.g., 30001. Then run:

python3 main.py --config configs/lego.txt

Please refer to Synthetic-NeRF dataset. Note to modify the dataset path in configs/lego.txt.

To include InfoInv, run below command:

python3 main.py --config configs/lego.txt --infoinv

To exclude InfoInv, run below command:

python3 main.py --config configs/lego.txt

The effectiveness of InfoInv for surface reconstruction is also proved in PET-NeuS.

• ✅ Learning UV mapping for texture editing.
• ✅ Learning Triplane projection for view synthesis.
• ✅ Including InfoInv in neural fields.
• ⬜️ Learning discrete transform for hash mapping.

If you use this code for your research, please cite our papers.

@inproceedings{zhan2023general,
  title={General Neural Gauge Fields},
  author={Zhan, Fangneng and Liu, Lingjie and Kortylewski, Adam and Theobalt, Christian},
  booktitle={The Eleventh International Conference on Learning Representations},
  year={2023}
}

Thanks NeuTex, K-Plane, TensoRF for releasing their implementation codes.