I have evaluated the depth model you gave ，why is it different? What might be wrong?
My device is 1080ti and the model for the evaluation is k_depth.tar, and the code used is your evaluation code

Is it fine to train all losses together without pre-trained model?
In my case of scratch training (simply tried ./train_resnet_256.sh), the photometric loss doesn't fall under around 1.0.

inverse_warp2
What is difference between projected_depth and computed_depth. Is the transformed depth map the projected depth?

Also, for projected depth why are you not using nearest neighbour interpolation. depthmaps must always use that right?

Also does this function doesnt transform the Z values between frames but keeps them?

I've noticed that the pose estimates, and their inverse (computed through compute_pose_with_inv) are consistently different from each other. It turns out that the predicted 'forward' translation (the third element of the pose prediction) is always larger when moving 'forward' in time, and smaller when moving 'backward' in time. That is, the predicted pose change from image t to t+1 is consistently larger than the predicted pose change from t+1 to t. You can reproduce this by adding the following lines to the train() or validate_without_gt() functions in train.py , and observing their values for a pretained model (I used the updated resnet18_depth_256 model) when training/evaluating the model:

print('forward', poses[0][:,2].mean().item(), poses_inv[0][:,2].mean().item()) print('backward', poses[1][:,2].mean().item(), poses_inv[1][:,2].mean().item())

Can you comment on why this might be the case? I would expect that between two images, the predicted forward translation should be the same magnitude, with one being positive and one being negative.

Hi, Have you ever visualized the valid mask gained by src_pixel_coords?

I find that the forward valid mask got by target image depth always is 0, which is wrong, the backward valid mask got by source image depth is right.
Looking forward to your reply!

Hi,

I was going through your paper and the algorithm looks great! I have a question about the geometry consistency loss, however.

Can you elaborate on the difference between this loss and the depth consistency loss in DF-Net. I'm talking about eq(7) in Section 3.4 (page 8).

Thanks!

I calculated the scale factor and scaled my predicted poses like original SfMLearner, and used the evaluation tool of Depth-VO-Feat (Zhan et al.)

scale_factor = np.sum(pred[:, :, -1] * gt[:, :, -1]) / np.sum(pred[:, :, -1] ** 2)
pred[:, :, -1] = pred[:, :, -1] * scale_factor

But the final result is so poor on KITTI seq09:
Average translational RMSE (%): 73.9656283885415
Average rotational error (deg/100m): 4.221253537085405

I used the same training hyper-parameters as in train_resnet_256.sh.

What should I do to fix it? Thanks!

Hi, I want to ask the reason why using median scaling during evaluation even though this method can predict scale-consistent depth of images. Is it necessary?

Hi, I've read your paper and the code and I want to ask you something about Depth loss.

When you compute photometric loss, you use depth weight mask to reduce the effects of geometric violations, right? But I want to ask the reason why this mask is unused to compute depth loss. I think it might work for both cases.

Thank you.

Hi,

Can you please share the script for demo of dense reconstruction using estimated depth?

Thanks

Hi, I have reconstructed point cloud with predict depth image, but there is something weird

so I reconstruct another point cloud with the true depth(calculated with lidar points)

I don't know what's wrong, could anyone give me some advice!
here are the point clouds
point_cloud.tar.gz

When I try to run run_depth_test.sh, it will use kitti data set to do evaluate. How can i convert predictions.npy to jpg file?

Hi, JiawangBian.
This work is pretty good, and I want to use it in other work if it can work under model of c++.
So if I want to implement this work in c++, what should I do?
I know I have to compile libtorch API , and convert model .th to .pt, but I don't have idea of modifying the python code to c++. Can you provide some advice?
Thank you very much!

Hi Jiawang,

I have 2 questions.

1. Why do you chose to use cpu in vo test rather than CUDA as we can see you set whether or not there is a CUDA judgment at the beginning of the program? But...in fact...
   

   SC-SfMLearner-Release/test_vo.py

   Line 63 in 078f5ae

   pose_mat = pose_vec2mat(pose).squeeze(0).cpu().numpy()

2. As u said in ur paper,

This is the first work to show that deep networks trained using monocular video snippets can predict globally scale-consistent camera trajectories over a long video sequence.

however I test vo with both left and right images, their trajectories are the same, and the scale is consistent with groundtruth, about 1705m and 1711m. However, when I set the left images as the trajectory reference, the Terminal output shows, both trajectories are only 1591 poses, 50.944m path length. What do you think about it?

1591 files to test
100%|█████████████████████████████████████████████████████████████████████████████████████████| 1590/1590 [05:29<00:00, 4.82it/s]

name: 09G
infos: 1591 poses, 50.944m path length

Hi，
When I started to run the test script, the following problem occurred, this is a small problem, but I did not debug it correctly, thank you
np.save(output_dir/'predictions.npy', predictions)
UnboundLocalError: local variable 'predictions' referenced before assignment

Hi @JiawangBian

Thanks for your great job. I am very honored to complete my research based on your framework. I will cite your work in my paper. Could you provide me the global pose data ( ORB-SLAM, zhou et al, zhan et al and yours ) in your figure 3? Thanks a lot.

Best regards,
Yu

Hi,

I ran pose inference using SFM-Learner's models (PyTorch) on sequence 10 and compared it with your models.

Using evo, this is the plot I get with global scale alignment:

However, in the paper, the plot comparing SFM Learner and your scale consistent version look different.

Do you have any ideas what I'm doing wrong?

Thanks,
Sai

Hi Jiawang @JiawangBian ,

How could you use the pre-trained model to do model finetuning? When I add the path of the pre-trained model to finetune_csk+k_resnet_256.sh, it outputs some errors below. Compared with
Clement Pinard's SfMLearner implementation, your train.py has changed something (just like output_writers)about the part. What do you think about it?

=> creating model
=> using pre-trained weights for PoseNet
=> using pre-trained weights for DispNet
=> setting adam solver
N/A% (0 of 200) | | Elapsed Time: 0:00:00 ETA: --:--:--
Traceback (most recent call last):
File "train.py", line 423, in
main()
File "train.py", line 190, in main
100% (472 of 472) |########################################################################| Elapsed Time: 0:00:00 ETA: 00:00:00
NameError: name 'output_writers' is not defined

Regards,
Yu

First of all, I want to congratulate you for your excellent work!
Lately, I have been trying a lot to make your model work on the Apolloscape dataset (particularly road01_ins dataset). With all the hyperparameters left as you propose, it seems that it cannot learn to infer quality disparity maps. Finetuning your already pretrained disparity and pose networks also seems to not work. Here is an example image:

The pretrained disparity network yields this result:

And after finetuning for 10 epochs:

Reducing the learning rate does not seem to make things better either. Is there something obvious I am missing? Are there any preprocessing steps that are mandatory? I am not doing any preprocessing except for reshape.

Thank you for your nice sharing, your code is very nice!

I found that you add "--with-gt" in train_resnet_256.sh. Is that mean using ground truth to validate the depth during training? Is that suitable for unsupervised learning?

I found that in the evaluation code, you still scale the output using the GT for every 5-frame snippet:

However in your work you claim the contribution of scale consistency depth, so I think you need to fix one global scale outside the loop and always use the same one, don't you? Am I misunderstanding something?

Also, I don't quite understand the statement you made in section 6.3,

Although the visual odometry results shown in the main paper is more important

in which sense you mean the results in the main paper are more important?
Thank you.

Your report is ：

But I use what you've provided repo to test is

I don't know why ？

Hi, I used test_vo.py to visualize the pose results of SfMLearner. There are some problems. Can you share the code to visualize the pose results of SfMLearner?

Hi @JiawangBian

I have a question about depth interpolation.

Here is the statement in your paper. You get the warped $D^{a}{b}$ by converting $D{a}$ a to 3D space and projecting to the image plane of $I_{b}$.$D'{b}$ is interpolated from $D{b}$. $D_{b} is the estimated depth maps of $I_{b}$. And you also implement this in your code. Note that we cannot directly use $D_{b}$ because the warping flow does not lie on the pixel grid.

In my mind, the depth in $D_{b}$ corresponds to each pixel of the original image. It does lie on the pixel grid. It does not require interpolation. In contrast, $D^{a}{b}$ does not lie on the pixel grid because it is warpped from $D{a}$. I think $D^{a}_{b}$ the one who needs interpolation.

After I read this paper by SfMLearner, I had such doubts. I wonder if this is the place where your thesis is inconsistent with the original. Looking forward to your opinion.

Regards,

Yu

Hi Jiawang @JiawangBian

I have a few puzzle about training.

1. I try set the sequence length at 6 (instead of 3), however its smooth loss and consistency loss decrease at first 10 epochs and then then keep increasing. Maybe It is overfitting? What's the meaning of the sequence length? What is its difference from batch size?

2. Have you try to use other batch size? Or in other words, 4 is the best batch size you have tried.

3. As we know, relative pose is from source image to target image plane. That means every 4 (batch size) images we can only get 3 relative pose. However your PoseNet outputs 4 poses every batch. I find you don't use the poes to warp the target image to ref_image in your code. It seems your PoseNet output is not the relative pose from source image to target image but it seems like the pose from World frame to camera frame. The same situation happen on test, which every image input could get a pose.

Best regards,

Yu

Hi, I am very interested your work, but I have some problem when recomplementing the results in the paper.

I runs the "Basic" method in following script.
python train.py $TRAIN_SET \ --dispnet DispNet \ --num-scales 1 \ -b4 -s0.1 -c0.0 --epoch-size 1000 --sequence-length 3 \ --with-mask False \ --with-ssim False \ --name posenet_256
I run the "Basic+SSIM" in the following script.
python train.py $TRAIN_SET \ --dispnet DispNet \ --num-scales 1 \ -b4 -s0.1 -c0.0 --epoch-size 1000 --sequence-length 3 \ --with-mask False \ --with-ssim True \ --name posenet_256
But the result is very terrible.
Could you tell me how you run the basic method?
Thanks.

Hi Jiawang,

I have two questions.

1. Scipy.imresize has not been only deprevated but removed entirely from scipy. Instead the devs recommend to use either numpy.array(Image.fromarray(arr).resize()) or skimage.transform.resize(). Could you please update the code to support Scipy-1.3.0?

2. How to run the visual odometry ? When I run the run_vo_test.sh(or test_vo.py), it only shows the trajectory result of 09 sequence (ground truth). I wanna know how to get the VO trajectory result of 09 sequence.

Kind regards,

Yu

Hello,thanks for this awesome project. I have the strange issue. I prepared my own dataset with imaqes 542x1024 and when training starts i always get
N/A% (0 of 200) | | Elapsed Time: 0:00:00 ETA: --:--:--
N/A% (0 of 946) | | Elapsed Time: 0:00:00 ETA: --:--:--
[torch.Size([2, 256, 34, 64]), torch.Size([2, 256, 34, 64])]
[torch.Size([2, 128, 68, 128]), torch.Size([2, 128, 68, 128])]
[torch.Size([2, 64, 136, 256]), torch.Size([2, 64, 136, 256])]
[torch.Size([2, 32, 272, 512]), torch.Size([2, 64, 271, 512])]
Dimension error when torch.cat(x,1)
Maybe its a stride , padding issue , please help me

Hi, when I start training with the pre-trained models (k_depth.tar & k_pose.tar), the initial Abs_Rel error on KITTI raw validation set begins with 0.09781. However, with the scratch training, the Abs_Rel error looks like below.

Is the Abs_Rel error with this scratch training eventually (200 epochs as the paper?) converge to the value of the pre-trained models?

Hi JIawang @JiawangBian

I'm confused with the valid_mask and the weight_mask. As you comment in the code, valid_mask means float array indicating point validity. But weight_mask looks like the real mask in your paper which could remove the dynamic object and occlusion. I looked over the #15 . Does valid_mask mean cropping the overlap part of 2 images to guarantee that the geometry consistency would not be disturbed by non-overlap parts?

Best regards,

Yu

Thank you for your great work, I am very interested in your work.
When I run your run_vo_test.sh to visualize odometry, I encounter such a problem---"evo_traj not found", could you help me figure out this problem?

Hi Jiawang,

Thanks for your effort in the updated version. I found some problems when I tried to train from scratch with default hyper-parameters. It looks overfitted after 3 epochs. Another question is that the GC loss looks conflicted with photometric loss at first.

flake8 testing of https://github.com/JiawangBian/SC-SfMLearner-Release on Python 3.7.1

$ flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics

./train.py:190:91: F821 undefined name 'output_writers'
            errors, error_names = validate_with_gt(args, val_loader, disp_net, 0, logger, output_writers)
                                                                                          ^
./train.py:192:104: F821 undefined name 'output_writers'
            errors, error_names = validate_without_gt(args, val_loader, disp_net, pose_net, 0, logger, output_writers)
                                                                                                       ^
./kitti_eval/depth_evaluation_utils.py:121:9: F821 undefined name 'LinearNDInterpolator'
    f = LinearNDInterpolator(ij, d, fill_value=0)
        ^
3     F821 undefined name 'LinearNDInterpolator'
3

E901,E999,F821,F822,F823 are the "showstopper" flake8 issues that can halt the runtime with a SyntaxError, NameError, etc. These 5 are different from most other flake8 issues which are merely "style violations" -- useful for readability but they do not effect runtime safety.

• F821: undefined name name
• F822: undefined name name in __all__
• F823: local variable name referenced before assignment
• E901: SyntaxError or IndentationError
• E999: SyntaxError -- failed to compile a file into an Abstract Syntax Tree

Hi, I have a question about the notation of depth maps (computed_depth, projected_depth) in 'inverse_warp2' function in 'inverse_warp.py'.

Suppose that D_a is ref_depth and D_b is tgt_depth as in your paper.
(Here the computed_depth and projected_depth are gonna aligned to tgt frame.)
In your paper, computed_depth is computed by warping ref_depth using relative pose. You didn't mention the tgt_depth in the paper, but we need it, right?

And, the projected_depth is computed by tgt_depth and relative pose.
(Actually, I am not clear about what that operation--interpolated depth(?) is...)
My main curiosity is that shouldn't we change the notations of those two variables (computed_depth and projected_depth) in the 'inverse_warp2' function?

Or, should I think that D_a is tgt_depth and D_b is ref_depth?

Hi,
I used the original data set downloaded by kitti, after running run_prepare_data.sh command to get the data set kitti_256, and then after train_resnet18_depth_256.sh, the following problems occurred. Is there something wrong with me? My system is ubuntu16.04 python=3.6 pytorch=1.2 cuda=10.
Thanks a lot

When I execute the sh scripts/run_inference.sh command, a depth map is generated under the result file, but these pictures are blank pictures. The following is the model I trained and the problems that occurred.
Hoping to get your help,
Thanks.

Are you sure that is the first work to show that deep networks trained using unlabelled monocular videos can predict globally scale-consistent camera trajectories over a long video sequence ???

Hi, I evaluated the depth using the method by running

sh scripts/run_depth_test.sh

But, I only got the depth predictions. So, I used the code based on Clement Pinard's SfMLearner implementation

test_disp.py

But, the depth results is not as good as the reported ones

  | Dataset | AbsRel | SqRel | RMS | RMSlog | <1.25 | <1.252 | <1.253
SC-Net repo | K | 0.137 | 1.089 | 5.439 | 0.217 | 0.830 | 0.942 | 0.975
SC-Net run | K | 0.140 | 1.100 | 5.481 | 0.222 | 0.824 | 0.940 | 0.974

The second row is our evaluation results. So, can you share the code for depth evaluation metrics such as AbsRel, SqRel, RMS, RMSlog and accuracy?

Thank you so much.

Hi Jiawang,

As we can see, there is a statement in your README.md.

and visualize the training progress by opening https://localhost:6006 on your browser.
However, it cannot be loaded. What's wrong with it？Maybe it is just a local link?

Best regards,
Yu

At line 108, I find that the result of set([subdir.basename()[:-2] for subdir in subdirs]) differ every time. I find that the data structure of Set in python3.6 is unordered. I want to know which python version you use or the result of this set is unordered in other ways.
Thanks.Looking forward to your reply.

Hi, when I evaluate the result using the method given by "Evaluation", I can't get the results like t_err, r_err. How can I get this two errors?

Thank you so much.

Thanks for your share of work. Could you please provide codes of testing on Make3d dataset? Thanks very much.

Hi, I trained the DispNet and PoseNet following the method in README, but the evaluation results were always worse than the reported ones. Can you give some advise? Thinks!

Our results are as follows:
  | Dataset | AbsRel | SqRel | RMS | RMSlog | <1.25 | <1.252 | <1.253
SC-Net repo | K | 0.137 | 1.089 | 5.439 | 0.217 | 0.830 | 0.942 | 0.975
SC-Net train | K | 0.148 | 1.226 | 5.741 | 0.230 | 0.805 | 0.934 | 0.972
SC-Net repo | CS+K | 0.128 | 1.047 | 5.234 | 0.208 | 0.846 | 0.947 | 0.976
SC-Net train | CS+K | 0.132 | 1.143 | 5.278 | 0.211 | 0.845 | 0.948 | 0.976

Hi, I want to train your model using IMU data without additional pose network.
So I found the code which can compute the pose of one frame, in

Then I want to figure out the way computing relative pose between two frames which can properly replace pose network. Would you please give me some advice to use those values of two frames for computing relative pose?

Hi Jiawang,
Best unsupervised implementation of monocular depth work, thanks for such wonderful code.
Just wanted your opinion.
I wanted to train sc-sfm with images captured using fisheye lens (190 deg H FOV). I looked at the sc-sfm code and I think the following three functions need change.
cam2pixel()
pixel2cam()
cam2pixel2().
Do you think it is sufficient?

When I run the script bash scripts/run_prepare_data.sh
I got

FileNotFoundError: [Errno 2] No such file or directory: Path('/Dataset/kitti_raw/2011_10_03/2011_10_03_drive_0058_sync/velodyne_points/data/0000000028.bin')

And there is no 0000000028.bin in that folder.
Have you ever met this problem?

I have a problem related to the file, it seems that there is a lack of cloud data, but I downloaded it from the official website again and found that the official website was already missing. Do you know what happened?

hi,I have a problem when run the script run_prepare_data.sh, it shows as follows

I don't know how to fix it, please give me some advice, thanks a lot!

Thank you for your sharing. But I can't open the link of pretrained model, could you please offer me another address?.

上述文件在什么地方下载？