alibaba / unifuse-unidirectional-fusion Goto Github PK

View Code? Open in Web Editor NEW

61.0 4.0 14.0 1019 KB

License: MIT License

Shell 0.23% MATLAB 13.24% Python 86.52%

unifuse-unidirectional-fusion's Introduction

UniFuse (RAL+ICRA2021)

Office source code of paper UniFuse: Unidirectional Fusion for 360$^\circ$ Panorama Depth Estimation, arXiv, Demo

Preparation

Installation

Environments

python 3.6
Pytorch >= 1.0.0
CUDA >= 9.0

Install requirements

pip install -r requirements.txt

Datasets

Please download the preferred datasets, i.e., Matterport3D, Stanford2D3D, 3D60 and PanoSUNCG. For Matterport3D, please preprocess it following M3D-README.md.

Training

UniFuse on Matterport3D

python train.py --data_path $DATA_PATH \
-dataset matterport3d \
--model_name Matterport3D_UniFuse \
--batch_size 6 \
--num_epochs 100 \
--height 512 \
--width 1024 \
--imagenet_pretrained \
--net UniFuse

Equirectangular baseline on Matterport3D

python train.py --data_path $DATA_PATH \
-dataset matterport3d \
--model_name Matterport3D_Equi \
--batch_size 6 \
--num_epochs 100 \
--height 512 \
--width 1024 \
--imagenet_pretrained \
--net Equi

It is similar for other datasets.

Evaluation

Pre-trained models

The pre-trained models of UniFuse for 4 datasets are available, Matterport3D, Stanford2D3D, 3D60 and PanoSUNCG.

Test on a pre-trained model

python evaluate.py  --data_path $DATA_PATH --dataset matterport3d --load_weights_folder $MODEL_PATH

Citation

Please cite our paper if you find our work useful in your research.

@article{jiang2021unifuse,
      title={UniFuse: Unidirectional Fusion for 360$^{\circ}$ Panorama Depth Estimation}, 
      author={Hualie Jiang and Zhe Sheng and Siyu Zhu and Zilong Dong and Rui Huang},
	  journal={IEEE Robotics and Automation Letters},
	  year={2021},
	  publisher={IEEE}
}

unifuse-unidirectional-fusion's People

Contributors

Stargazers

Watchers

Forkers

siidej pean1128 3000huyang benjamesbabala 121644048 highclow halleyjiang ajunlonglive nivesh48 arctanbell zhijieshen-bjtu zebrajack jaychen125 clarkren

unifuse-unidirectional-fusion's Issues

Pretrained Model

Hi, thanks your great work, but how can I access the pretrained model？Is there any link?

Thank you very much for your work, but there is a piece of code that makes me a little confused.
The location is Line18-Line21 in metrics.py
I understand that it means that gt and pred are limited to the range of 0.1-10 before calculating the evaluation metrics. However, the logic of these four lines only limits gt to 0.1 to 10, while pred will have an area greater than 0.1.
I hope you can remove my confusion.

train.py

Coud u tell me pls. Whats wrong with this, after run python train.py I have error.

How to get the 3d60 train data set

When I use 3d60（Left/Down）to evaluate, it's OK, but some data will be missing during training.

Data Split

It seems that your data split is different from OmniDepth and BiFuse.
How do you decide to use this split?

PanoSUNCG

How to download PanoSUNCG? I do not see any link in https://fuenwang.ml/project/360-depth/.

Thanks for your help. Looking forward for the reply!

The quantitative results in the paper cannot be reproduced

Hi, I have tested the Stanford2D3D dataset with the given model parameters, but the results are vary different with the quantitative results in the paper. Have you uploaded the wrong model parameters? The results with the given parameters are

Besides, I have tried to retrained the network on Stanford2D3D with PyTorch of the same version with your codes, but the results are worse than that in the paper too. My reproduced results are

Is there any difference between the given codes and the implementation used in the paper?

Multiple GPU bug

When I set --gpu_devices as [0, 1, 2, 3], the training process can only work on single GPU. If any other setting should be conducted? Thanks!

The problem about testing results using the 3D60 dataset

Hello, I encounter a problem similar to zcq15 about 3D60 dataset test. I have tryed to retrained the network on 3D60, but the results seemed to poor compared with the results in paper. I just used your codes directly and didn't do any data preprocessing on 3D60. The results with the given parameters are

I have also tested the 3D60 dataset with the given model parameters, but it seem to worse than the results in paper too. the results are below

Is there any additional operations on 3D60 dataset just like Stanford2D3D?
Thanks for your help. Looking forward for the reply!

How to obtain GPU memeory of network?

Hi, thanks for your excellent work. I wanted to ask how the GPU memeory is obtained or calculated in this paper ?

How to deal with Matterport3D images

Hi,

Can you please tell us how to handle Matterport3D images? I mean, once you downloded the dataset from the official website, how did you create the equirectangolar RGB images and depth maps?

Thanks in advance,
Valerio

About Datasets

For the MatterPort dataset, there are 1TB files on the official website. I don't know which parts I need to download to complete this; For the 3D60 data set, there are also hundreds of GB; For PanoSUNCG, the official website did not provide a download link, and the author did not reply; For standfor2D3D, the official website maintenance also cannot obtain data sets. I wonder if you can share the dataset, thank you!

evaluate.py

Hi nice work.
Coud u tell me pls. Whats wrong with this, after run python evaluate.py I have error.

python evaluate.py --data_path /Users/kgkozlov/Downloads/img_4.png --dataset matterport3d --load_weights_dir /Users/UniFuse-Unidirectional-Fusion/PretrainedModels/

File "/UniFuse-Unidirectional-Fusion/UniFuse/datasets/matterport3d.py", line 83, in __getitem__
    rgb = cv2.cvtColor(rgb, cv2.COLOR_BGR2RGB)
cv2.error: OpenCV(4.5.3) /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-req-build-z9mn802i/opencv/modules/imgproc/src/color.cpp:182: error: (-215:Assertion failed) !_src.empty() in function 'cvtColor'

fpath = os.path.join(os.path.dirname(__file__), "datasets", "{}_{}.txt")

But you add txt file with RGB and DEPTH, But shouldn't we just add one RGB image ?

Is there anything about Matterport Preprocessing

Download PanoBasic

Copy stitching_Matterport3D.m to PanoBasic

Modify the directories in stitching_Matterport3D.m, i.e., changing source_dir to the folder of Matterport3D and changing target_dir as the output folder of panorama images and depth maps.

Execute stitching_Matterport3D.m using Matlab.

It seems that stiching_Matterport3D uses stich.m, while stich.m is missing in PanoBasic.

@robertsheng

How to deal with Stanford2d3d?

In the original version, the top and bottom parts of stanford2d3d are missing, but in your paper, it is full. How to deal with it?

About 'cube_inputs' for unifuse.py.

[Question]
cube_inputs = torch.cat(torch.split(input_cube_image, self.cube_h, dim=-1), dim=0) in unifuse.py.
this code seemingly cat six cube face on dim0. i.e., (batch_size, 3, h, 6h) --> (batch_size6, 3, h, h).
I think that the cube encoder can not ultize feature of different minibatch. Intuitively, catting these face on dim of channel are responable. But, it can't use the imagenet-pretrained network.
Did I understand correctly?