yuhsuanyeh / bifuse Goto Github PK

w/o --nocrop

with --nocrop

Message when running this on Ubuntu 18.04

Test Data Num: 2
Load: BiFuse_Pretrained.pkl
Validation process:   0%|                                                                                            | 0/2 [00:00<?, ?it/s]/home/gateway/anaconda3/lib/python3.7/site-packages/torch/nn/functional.py:2693: UserWarning: Default grid_sample and affine_grid behavior will be changed to align_corners=False from 1.4.0. See the documentation of grid_sample for details.
  warnings.warn("Default grid_sample and affine_grid behavior will be changed "
Validation process: 100%|████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:03<00:00,  1.61s/it]

Thanks for providing the tools to convert this into a point cloud but it's limited to really only python programmers or Open3D. I use a free tool CloudCompare https://www.danielgm.net/cc/ or Meshlab to work with my point clouds. Is their any chance to convert to .ply format or any of the ones listed in this screen shot?

Any chance to get request/feature tags ?

Hmm, I'm used to having depth maps where white is closer and black or grey areas further back. Did I miss something or are their 2 ways to handle this?

Yours..

quick google image search..

I would like to ask the Dataset Usage and the Experiment Settings of the pretrained model provided.
And would the training/testing codes in the paper released in the near future?
Thx!

Hi there,

I think this project is very interesting!
I was wondering if it is in theory possible to run it from a live video feed in realtime to produce something like a depth map at 5hz?

Thanks in advance!

Cheers,
Tom

Hi, I want to train BiFuse model on a new dataset, but this repo contains no training code, such as loss function, metrics and training script, etc. So could you please kindly make the training code public? Thanks a lot.

Hi, I am trying to determine what conversion to use to transform the depth matrix into distance (meters from the camera axis to the respective x,y pixel coordinate).
How would you recommend I go about the conversion to actual distance given the depth map?
Example: on the sample image min_depth is 0, max_depth is ~9.06, median_depth is 1.67. Some far away object in the image (probably 10m in real life) has a depth of 1.86.

In other papers they just normalize by using the max observed distance in the training data; I have no idea what that would be in your training data set or if that would be correct to do in this case.
Thank you!

Excuse me, can you tell me how much room synthetic? This is just a anoramic view of a single point, multiple room what method?

Hello!

I'm trying to run your script but run into memory problems. I'm not sure how to tackle this; I tried to use a smaller sample image, but even at size 100x50 I still run out of memory. Maybe it is not related to the image size?.. The error messages I get are quoted below, and the numbers there don't seem to be related to the image size. If I look at the amount of gpu memory used as the script runs, it starts at more or less zero and increases to the max of 2048Mb then the script exits..

Any suggestions are welcome!

thnx.

Test Data Num: 1
Load: BiFuse_Pretrained.pkl
Traceback (most recent call last):
File "main.py", line 115, in
main()
File "main.py", line 111, in main
saver.LoadLatestModel(model, None)
File "/sda1/bifuse/BiFuse/Utils/ModelSaver.py", line 33, in LoadLatestModel
params = torch.load(name)
File "/home/jos/.local/lib/python3.6/site-packages/torch/serialization.py", line 608, in load
return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
File "/home/jos/.local/lib/python3.6/site-packages/torch/serialization.py", line 787, in _legacy_load
result = unpickler.load()
File "/home/jos/.local/lib/python3.6/site-packages/torch/serialization.py", line 743, in persistent_load
deserialized_objects[root_key] = restore_location(obj, location)
File "/home/jos/.local/lib/python3.6/site-packages/torch/serialization.py", line 175, in default_restore_location
result = fn(storage, location)
File "/home/jos/.local/lib/python3.6/site-packages/torch/serialization.py", line 155, in _cuda_deserialize
return storage_type(obj.size())
File "/home/jos/.local/lib/python3.6/site-packages/torch/cuda/init.py", line 606, in _lazy_new
return super(_CudaBase, cls).new(cls, *args, **kwargs)
RuntimeError: CUDA out of memory. Tried to allocate 14.00 MiB (GPU 0; 1.96 GiB total capacity; 1.25 GiB already allocated; 15.56 MiB free; 1.28 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

Hi! We observed some discrepancy between the code and the paper regarding the inputs that go through the Refine() module. On the paper (according to Figure 4.) the input is a concatenation of depth estimations comming from the 2 branches (Equirectangular and projected Cubemap) making it a 2 channel Tensor.

But on the code, the concatenation additionally includes the RGB input tensor making it a 5 channel Tensor mixing color and depth information:
https://github.com/Yeh-yu-hsuan/BiFuse/blob/6fb1cbe8a3c3891a9067f595ba2af9d14f8ae1c6/models/FCRN.py#L522

Which one of the 2 approaches is the correct one?

Also, on the last projection to bring the Cubemap branch output to Equirectangular, the transformation is done in 2 steps:
https://github.com/Yeh-yu-hsuan/BiFuse/blob/6fb1cbe8a3c3891a9067f595ba2af9d14f8ae1c6/models/FCRN.py#L519-L520

Couldn't the transformation self.ce.C2E() be directly applied to the Cubemap branch output (as the Fusion blocks do)?

Is their a way that we can have an option or maybe I'm missing it where it doesn't crop/scale down the size? I have images that are 7296 x 3648 which comes out of my Theta Z1 360 camera.

Hello,
I am working on something similar and I want to know how long it took you to train this entire network?

Hi, I am implementing the loss described in your paper, in equation (5) (reverse Huber Loss). Would it be possible to know what value are you using for the c parameter in equation (4)?

Hi, I have a question about pretrained model.
It seems the pretrained model provided is used for Matterport3D dataset.
Could you please share other pretrained models for Stanford2D3D and 3D60.
Thanks!

Hey! I find the paper doesn't show loss function in training stage completely, it just introduce reverse Huber loss for optimizing predictions from both Be and Bc, what about the next two stage?

Can I run the code without GPU on mac. If it can be run without GPU then please tell the steps?

The visualization is implemented in vis3D.py, I try to save it, but the points in resulting save ply file is all black, at the using meshlab

Hello, I want to try the network on Matterport3D, but experience problems when stitching views. My results are shown in the picture below. Could you please provide me your stitching codes or preprocessed dataset for Matterport3D? Thanks!