visinf / self-mono-sf Goto Github PK

I have trained the model on my custom data with 5000 stereo pairs and 20 epochs.I got the following results
Input image

Output disparity: gray and colored respectively as shown:


Flow:

What can be the reason for this?
Could this also happen if the correct focal length is higher by 20 units as compared to that given for training?
Any suggestion is appreciated!
Thanks!

Initially ,thank you so much for your code, for your answers and help .
could you please explain why do you call this method self-supervised ? could you please point out the specific point that let you call it self-supervised?
Thank you again ..

Hi @hurjunhwa,

I am just wondering have you tested the compatibility of correlation package with Pytorch >=1.6.0? For Pytorch >=1.6.0, the cuda version supported is 10.1 & 10.2, not 10.0. If I used torch == 1.6.0+cu101 & cuda 10.1. It gives follows error in runtime:

File "/disk_ssd/self-mono-sf/models/correlation_package/correlation.py", line 4, in
import correlation_cuda
ImportError: libcudart.so.10.0: cannot open shared object file: No such file or directory

For this error, I am not sure why in correlation_cuda will try to find libcudart.so.10.0 instead of libcudart.so.10.1

If using torch == 1.6.0+cu101 & cuda10.0, it gives this error in runtime:

File "/disk_ssd/self-mono-sf/models/correlation_package/correlation.py", line 4, in
import correlation_cuda
ImportError: /home/hashswan/.conda/envs/self-mono/lib/python3.7/site-packages/correlation_cuda-0.0.0-py3.7-linux-x86_64.egg/correlation_cuda.cpython-37m-x86_64-linux-gnu.so: undefined symbol: _ZN6caffe26detail37_typeMetaDataInstance_preallocated_32E

For this error, it resulted from incompatibility of pytorch version and cuda version

In both cases, package installation is successful without any warnings.

I am looking forward to your reply. Thank you very much!

Best,

Thank you for this great work!
I am a bit confused on the data directory structure for running experiment. Is this the same as that in MonoDepth?
https://github.com/OniroAI/MonoDepth-PyTorch#dataloader

I am looking forward to your reply.
Thank you!

Hello great work @hurjunhwa and team !
I was able to run the test code and got the disparity and optical flow images for kitti dataset.I wanted to visualize the scene flow.How do I do that using the disparity and flow?
Also how to save the 3d motion vector for every test image?
I want to predict the speed of the objects using the 3d motion vector given by the scene flow.To do so,
I just need to take the difference of the two 3d positions in two consecutive frames and divide it by the time interval right?
Does the predicted motion vector require any further scaling?
Any suggestion is greatly appreciated!
Thank you.

Hello,

I have encountered a memory problem during training. Could you specify your hardware specifications for training?

Thank you

Hi ,
First I'd like to thank you for this great project !
I wanna ask about the relation between the loss values (D1-all D2-all F1-all SF1-all) you have mentioned in Table 4 in the paper which reports the " Monocular scene flow evaluation on KITTI Scene Flow Training" , and the losses that your code produces ( image reconstruction loss, point reconstruction loss, 3D smoothness loss, scene flow loss, disparity loss and total loss) . I just wanna get the same losses as you reported in your paper. could you explain how to get these loss values using your code please ?
Thanks in advance

Hi, this is a really interesting project! Will there be code for running this network on any dataset for testing and training? If not, how would you recommend approaching this?

Cheers

Hi,I want to run this code on 3090,but the 3090 needs to install CUDA11.0.When I run ./train_monosf_selfsup_kitti_raw.sh,error:ImportError: libcudart.so.10.0:cannot oprn shared object file:No such file or directory.
So, I would like to ask how to solve it? I learned what changes to make to the code.Looking forward to your answer.

Hello,

I am trying to run train_monosf_selfsup_kitti_raw.sh file but I have encountered an error.

No module named 'colorama'

In the closed issues, I have seen that tqdm version causes a problem that's why I don't want to install needed packages without knowing which version you used. Could you provide a requirements.txt?

Thank you

Hello, Sorry to interrupt you. I am interested in your work. But I meet a problem when I run 'train_monosf_selfsup_kitti_raw.sh'.

Traceback (most recent call last):
File "../main.py", line 90, in
main()
File "../main.py", line 87, in main
validation_augmentation=validation_augmentation)
File "/data/self-mono-sf-master/core/runtime.py", line 455, in exec_runtime
total_progress = create_progressbar(**progressbar_args)
File "/data/self-mono-sf-master/core/runtime.py", line 92, in create_progressbar
return tools.tqdm_with_logging(**tqdm_args)
File "/data/self-mono-sf-master/core/tools.py", line 398, in tqdm_with_logging
logging_on_update=logging_on_update)
File "/data/self-mono-sf-master/core/tools.py", line 344, in init
postfix=postfix)
File "/data/my_anaconda3/envs/self_mono/lib/python3.7/site-packages/tqdm/std.py", line 1109, in init
self.refresh(lock_args=self.lock_args)
File "/data/my_anaconda3/envs/self_mono/lib/python3.7/site-packages/tqdm/std.py", line 1361, in refresh
self.display()
File "/data/my_anaconda3/envs/self_mono/lib/python3.7/site-packages/tqdm/std.py", line 1509, in display
self.sp(self.str() if msg is None else msg)
File "/data/my_anaconda3/envs/self_mono/lib/python3.7/site-packages/tqdm/std.py", line 1165, in str
return self.format_meter(**self.format_dict)
TypeError: format_meter() got an unexpected keyword argument 'nrows'
Exception ignored in: <function tqdm.del at 0x7f2c0ed42f80>
Traceback (most recent call last):
File "/data/my_anaconda3/envs/self_mono/lib/python3.7/site-packages/tqdm/std.py", line 1162, in del
self.close()
File "/data/self-mono-sf-master/core/tools.py", line 374, in close
if self._logging_on_close and not self._closed:
AttributeError: 'TqdmToLogger' object has no attribute '_logging_on_close'

Hi, thanks a lot for the excellent work.

I have a question why does the sequence occlusion mask include the stereo mask? Thank you!

https://github.com/visinf/self-mono-sf/blob/eec356d95038da49e6705194e6dc0780b750f2b0/losses.py#L205C8-L205C8

Hi,

Thanks for sharing the great work. I found the link to the Eigen Split Projected Depth for the monocular depth evaluation on the Eigen Split in the readme is not valid. Could you check it out and share a valid link to the depth data? Thanks!

Hi, I am very interested in your work!

I reimplemented your code on nuscenes and I noticed the line 134 in the function projectSceneFlow2Flow that the pixelgrid is not normalized. It makes the obtained flow to be big (absolute value), which results in a bad visualization. I really want to know whether it should be normalized.

Looking forward to your reply. Thank you!

Hi,
after executing the evaluation script ./eval_monosf_finetune_kitti_test.sh, the tqdm output keep printingsf=0.0000.

But the depth evaluation works well. Any suggestions?

Hi @hurjunhwa those configs seem not be used in main.py

--training_dataset_root=$KITTI_RAW_HOME \
--training_dataset_flip_augmentations=True \
--training_dataset_preprocessing_crop=True \
--training_dataset_num_examples=-1 \

Hello @hurjunhwa and team!
I am using this code to train on my own dataset.In regard to this I had a few questions as follows:
I wanted to know how did you get the scaling of 0.54 in the following code:

Hi ,
I got this error while I am trying to compile the warping_package

how could I handle this error please
thanks in advance

Which is your tqdm version, error occur!!!

Traceback (most recent call last):
  File "../main.py", line 90, in <module> 
    main()  
  File "../main.py", line 87, in main  
    validation_augmentation=validation_augmentation)  
  File "/home/administrator/SF/mono-sf/core/runtime.py", line 455, in exec_runtime  
    total_progress = create_progressbar(**progressbar_args)  
  File "/home/administrator/SF/mono-sf/core/runtime.py", line 92, in create_progressbar  
    return tools.tqdm_with_logging(**tqdm_args)  
  File "/home/administrator/SF/mono-sf/core/tools.py", line 406, in tqdm_with_logging  
    logging_on_update=logging_on_update)  
  File "/home/administrator/SF/mono-sf/core/tools.py", line 344, in __init__  
    postfix=postfix)  
  File "/home/administrator/virtualenv/py3.6_pytorch1.3/lib/python3.6/site-packages/tqdm/std.py", line 1050, in __init__  
    self.refresh(lock_args=self.lock_args)  
  File "/home/administrator/virtualenv/py3.6_pytorch1.3/lib/python3.6/site-packages/tqdm/std.py", line 1338, in refresh
    self.display()
  File "/home/administrator/virtualenv/py3.6_pytorch1.3/lib/python3.6/site-packages/tqdm/std.py", line 1471, in display
    self.sp(self.__repr__() if msg is None else msg)
  File "/home/administrator/virtualenv/py3.6_pytorch1.3/lib/python3.6/site-packages/tqdm/std.py", line 1089, in __repr__
    return self.format_meter(**self.format_dict)
TypeError: format_meter() got an unexpected keyword argument 'nrows'
Exception ignored in: <object repr() failed>
Traceback (most recent call last):
  File "/home/administrator/virtualenv/py3.6_pytorch1.3/lib/python3.6/site-packages/tqdm/std.py", line 1086, in __del__
    self.close()
  File "/home/administrator/SF/mono-sf/core/tools.py", line 374, in close
    if self._logging_on_close and not self._closed: 
AttributeError: 'TqdmToLogger' object has no attribute '_logging_on_close' 

Hi!
Nice code, while would you like to elaborate more on this piece of code? What the post-processing for, data augmentation?

flow_f_pp.append(post_processing(output_dict['flow_f'][ii], flow_horizontal_flip(output_dict_flip['flow_f'][ii]))) #?
flow_b_pp.append(post_processing(output_dict['flow_b'][ii], flow_horizontal_flip(output_dict_flip['flow_b'][ii])))
disp_l1_pp.append(post_processing(output_dict['disp_l1'][ii], torch.flip(output_dict_flip['disp_l1'][ii], [3])))
disp_l2_pp.append(post_processing(output_dict['disp_l2'][ii], torch.flip(output_dict_flip['disp_l2'][ii], [3])))

I notice that for every layer except, the disparity is limited to 0-0.3 by “self.sigmoid(disp_l1) * 0.3”. Why is it designed this way？I know the ground truth of the disparity is limited to 0-1 by the function read_png_disp in common.py. I think it may not be appropriate for other datasets such as Flyingthings3D because normalization is difficult to conduct as disparity is not between 0 and 255. If not normalization, will the result be worse?

@hurjunhwa
Hi ! Thanks for your great work for the visualization code of scene flow!
But it seems that I've got one problem with it.

As you may see in the picture, I didn't change any of the data except for the three directories:kitti_data_dir, result_dir, vis_dir
I want to know where the problem may occur.
Is it the path problem or is the installation version of open3d different?

Thanks for making your code open,

Can you tell me how do you obtain [0.93, 1] scaling for size [256, 832] in your Augmentation? Is scaling dependent on the resized image?