I was reading your paper "On the uncertainty of self-supervised monocular depth estimation". Firstly, I would like to congratulate you on writing such an useful and good paper. Not much research has been done on the Uncertainty topic, and you have done it. Thank you for that. I have a question regarding the loss function in the Self Teaching approach. Since the training code is not provided, I wanted to try it out myself as I wanted to reproduce your results for my project. If my understanding is correct, I first use the pretrained model to generate pseudo ground truths, and then use them to train the Student Network. The student network has two channels as outputs, one is disparity and the other is the uncertainty. So, in the loss function ,

1. Does  mean the disparity of the Student network and   mean the uncertainty generated at the second output channel.
2. Does the L1 difference operator take pixel wise difference between Teacher and the Student and finally the sum of all the pixel-wise losses are taken to minimize? 
   Please let me know. I am stuck at this point and it would be really helpful if you could shed some light on it. 

Edit : I have already seen the issue #2 , there was no real mention of Loss there. This question might be a simple one, but I am no expert in this field, I have just started working on this. Excuse any ignorance. Thanks.

Hi, thanks for your great work. I noticed that there were two work for MDE in CVPR20 using uncertainty loss, another work was D3VO. Both of you used the same uncertainty loss (log section in your paper), but gotten totally different uncertainty map. I can get uncertainty map as yours. So I‘d like to ask if you know the reason. Looking forward to your reply. Thanks.

Hi, you mentioned that we can decouple depth and pose when modelling uncertainty with self-teaching strategy in the paper. But I cannot figure out why this scheme can provide appropriate uncertainty.

For example, if our teacher model is poorly trained and provides inaccurate depth estimation, with self-teaching we can still get depth uncertainty from student model. In this situation, can we trust this uncertainty?

Hey @mattpoggi ,

I understand that the bayesian uncertainty is calculated by adding the uncertainty from a predictive method and an empirical method, for example snap+log. However, I seem to be missing something.

In line 183, you are adding the variance from the empirical method and adding up the uncertainties from the predicitive method(log).

However in line 216, you are taking the exponential for the log method, I believe this is because the uncertainties are in fact the \log of uncertainities.

Why do you not use the same torch.exp on the uncertainties while calculating it in the bayesian way, in line 183 above. What am I missing here?.

Thanks in advance

When I am reproducing the results, I got v. wrong results, as shown below:

So when I checked plotting the depths and uncertainties saved in raw/disp and raw/uncert then they are like this (see image below, which is clearly wrong):

The thing is I am not getting proper raw pred_depths and uncertainties in raw directory, however my qualitative pred_depths and uncertainties are saved correctly in qual directory as shown below some examples:

So the main issue is when we are scaling the raw pred_depths and uncertainties before saving.

1. Could you please tell why we are multiplying by gt_depth's width in this line 270 of generate_maps.py: when you are saving the depths, why are you scaling it like this:

pred_disps[i]*(dataset.K[0][0]*gt_depths[i].shape[1])*256./ratio/10.

(I know depth = (baseline*focal_length)/disparity) but why by ratio, 256, 10 and gt_depth[i].shape[1] which is the width of the image.

2. And when saving uncertainties in line 279, why scaling like this: pred_uncerts[i]*(256*256-1)

3. So when using these saved maps in evaluate.py you are again scaling pred_depths like this before comparing with gt_depth: I can't understand line no. 167.
   where you are dividing pred_disp like this: src = pred_disp / 256. / (0.58*gt_depth.shape[1]) * 10

Please help for this issue. @mattpoggi @FilippoAleotti @fabiotosi92

Hi,

The line here is supposed to sort all the uncertainties from high to low right? But here we are simply negating it, how does this sort it in the reverse order?

Your paper also states the same Given an error metric, we sort all pixels in each depth map in order of descending uncertainty.

Regards,
Shrisha

Hi,

I want to apply your self-teaching approach to a different dataset and thus have to train it myself. In order to make sure that I get everything right, I am trying to reproduce your results on KITTI first, but so far without success.

First, I generated the teacher ground-truth using a modified test_simple.py, which stores the predicted depth for all 4 scales as .npy files:

python test_simple.py --image_path KITTI_RAW_DATA_PATH --image_file splits/eigen_zhou/train_files.txt --out_folder KITTI_TEACHER_DATA_PATH --model_name mono_640x192 --pred_depth

(Repeated the same with val_files.txt, of course.)

I then trained the student network using the following loss function:

def compute_loss(self, inputs, outputs, use_sigmoid=True):

    final_loss = 0

    losses = {}

    for scale in range(self.num_scales):

        depth_teacher = inputs[("pred_depth", scale)]

        _, depth_student = layers.disp_to_depth(outputs[("disp", scale)], self.min_depth, self.max_depth)

        if use_sigmoid:
            sigma = torch.sigmoid(outputs[("uncert", scale)]) + 1e-6
            log_sigma = torch.log(sigma)
        else:
            log_sigma = outputs[("uncert", scale)]
            sigma = torch.exp(log_sigma) + 1e-6

        l1_loss = torch.abs(depth_student - depth_teacher.cuda())
        loss = l1_loss / sigma + log_sigma

        losses[("l1_loss", scale)] = torch.mean(l1_loss)
        losses[("loss", scale)] = torch.mean(loss)

        final_loss += losses[("loss", scale)]

    final_loss /= self.num_scales

    losses["loss"] = final_loss

    return losses

With use_sigmoid=False, I get NaN loss in the last epoch. Evaluation of the second to last checkpoint (weights_18) on eigen_benchmark yields:

   abs_rel |   sq_rel |     rmse | rmse_log |       a1 |       a2 |       a3 | 
&   0.136  &   1.300  &   4.833  &   0.186  &   0.857  &   0.955  &   0.981  \\

   abs_rel |          |     rmse |          |       a1 |          | 
      AUSE |     AURG |     AUSE |     AURG |     AUSE |     AURG | 
&   0.029  &   0.062  &   0.497  &   3.583  &   0.032  &   0.090  \\

With use_sigmoid=True, training finishes without a problem and the last checkpoint (weights_19) yields:

   abs_rel |   sq_rel |     rmse | rmse_log |       a1 |       a2 |       a3 | 
&   0.131  &   1.200  &   4.769  &   0.184  &   0.857  &   0.958  &   0.983  \\

   abs_rel |          |     rmse |          |       a1 |          | 
      AUSE |     AURG |     AUSE |     AURG |     AUSE |     AURG | 
&   0.029  &   0.058  &   0.491  &   3.541  &   0.034  &   0.088  \\

This is still far from what I get with your pre-trained network. Any idea where the problem might be?
I am using PyTorch 1.10.1 instead of 0.4, since that old version is not compatible with newer GPUs, but otherwise I am unable to find differences to what you have described in your paper and published code. Training the original Monodepth2 with the newer PyTorch version also gives nearly identical results to the published ones (e.g. AbsRel 0.090 -> 0.091, RMSE 3.942 -> 3.993), so I doubt that this would be a problem.

Thanks!

Hi There,

I'm wondering if there is a discrepancy between this line of code and equation 17 in the paper that computes the full uncertainty for the Bayesian approach:

As far as I'm aware, the empirical variance from the ensemble should be added with the square of the learned uncertainty output. Am I correct in associating this line of code with equation 17 of the paper? If so, there might be an issue here. I was wondering if this affects the visualization of uncertainties in the paper.

Thanks!

Hi, @mattpoggi,
Thanks for your excellent work!!
I have some questions about my implementation.

1. In the depth decoder (S), it is output 4 scale disparity and uncertainty, I would like to know that have you calculated all scale outputs when you implemented eq.14?
   

2. If my uncertainty has some negative values, Should I limit it to>0?

Hi, thanks for your greate work! And I am confused about the dataset split in training, can you tell me which split files were choosed when you training the model? is 'eigen zhou' ? or 'benchmark'? or others. it is my first time to use kitti_dataset. Thanks!

Thx for your nice work! I wonder how to train the self-teaching network. It seems you didn't give the code for training. According to (14), a group of results is needed, will it takes a lot of GPU memory during training?
Besides, I didn't find the drop+self-teaching result in the paper, I wonder the performance of this way.

Hi, Matteo! Will the training code be published in the future?

Hey @mattpoggi ,

I tried to look for initial learning rate for the "snap" training mode, but I could not find it. A couple of questions related to snap,

1. The original Snap authors use initial learning rates of 0.2 and 0.1, Did you also use 0.2 as the initial learning rate?
2. You use "C" as 20 meaning one Snap every epoch and choose last N Snaps or N random Snaps? (N being 8 according to your paper), Also T = total number of training iterations , which means for eigen zhou split it would be (39810//batch_size)* num_epochs?

Thanks in advance 👍

Thanks for your excellent works!!!

Hey @mattpoggi ,

I was trying to train the log model. I made necessary changes to the decoder to include the additional channel. When I start training, the intial loss is NaN and then after some batches it is NaN again. I was debugging the issue and stumbled upon this piece of code from your decoder.py

1. In line 81, sigmoid is used as the original code from monodepth2, but I do not see sigmoid being used for uncerts in line 85, Is there any reason for this?

2. I train on the GPU, but for debugging I use the CPU. While debugging on my CPU with batch_size 2 (any size greater will cause memory issues), I used breakpoints to see the values of uncert.

As seen in the image, the min value is negative, Log of a negative number is
NaN. This made me ask the first question, why the uncerts are not clamped
between 0(possibly a tiny bit greater to avoid inf when log is taken in the
loss function) and 1.
Is my understanding right or have I misunderstood something?

3. My loss function is
   

   Is there a problem with this? Do you also use the "to_optimise" which is a
   min(reprojection_losses and identity losses) or just the original reprojection
   losses?

EDIT : After reading quite a lot, I feel that my log loss is wrong. Maybe the uncertainities coming at the output channel are already \log(uncertainties) , so I would have to correct my loss function to below?

EDIT 2: Would the above edit hold good for the self teaching loss too, meaning the uncertainity outputs are actually the \log(uncertainties), so I have to take torch.exp() in the loss.?

Thanks in advance

Hi, Matteo! Can you share the training code to me if you don't mind it?

Hi!
When I try to increase the uncertainty part (described as the log section in your paper) in the network, the loss changed to negative.
Is this normal？ How did you solve it?

Hi,
Thanks for this great work. In Eq. 12/13 you have a log(variance) that might go negative for variance<1. Assuming, variance to be always +ve can we modify the Eq. 12/13 to log(1+variance) to prevent negative loss and allowing better optimization?
Will this modification hamper training or net output during inference?

When I use monodepth2 and implement self-teaching, uncertainty does improve the results. But why I use a lightweight network and implement self-teaching, the result is not better but worse?