Code for "Unsupervised Adaptation for Deep Stereo" - ICCV17

License: Apache License 2.0

C++ 88.50% Cuda 11.26% Shell 0.24%

computer-vision stereo-vision deep-learning machine-learning adaptation disparity-map iccv17 iccv iccv-2017 confidence

unsupervised-adaptation-for-deep-stereo's Introduction

Unsupervised-Adaptation-for-Deep-Stereo

Official Code for "Unsupervised Adaptation for Deep Stereo" - ICCV17 (http://openaccess.thecvf.com/content_ICCV_2017/papers/Tonioni_Unsupervised_Adaptation_for_ICCV_2017_paper.pdf)

This is the original caffe implementation used for the ICCV17 paper, since then we have also developed an easier to handle tensorflow implementation available here: https://github.com/AlessioTonioni/dispflownet-tf. (DISCLAIMER: the performance of the Dispnet trained on tensorlfow are slightly worse than those of the original one, act accordingly)

This code is intended to be plugged into DispNet by Mayer et al., available at https://lmb.informatik.uni-freiburg.de/resources/binaries/dispflownet/dispflownet-release-1.2.tar.gz If you use this code, please cite our paper:

@InProceedings{Tonioni_2017_ICCV,
  author = {Tonioni, Alessio and Poggi, Matteo and Mattoccia, Stefano and Di Stefano, Luigi},
  title = {Unsupervised Adaptation for Deep Stereo},
  booktitle = {The IEEE International Conference on Computer Vision (ICCV)},
  month = {Oct},
  year = {2017}
}

List of source files:

/dispflownet-adaptation-release/src/caffe/layers/custom_data_layer: Allows to read confidence in addition to stereo pairs and disaprity ground-truth.
/dispflownet-adaptation-release/src/caffe/layers/l1loss_layer: Includes the confidence weighted L1 loss used for unsupervised adaptation.
/dispflownet-adaptation-release/src/caffe/proto/caffe.proto: Definition of the confidence data type CONF
/dispflownet-adaptation-release/tools/convert_imageset_disparity_confidence.cpp and convert_imageset_disparity_confidence_from_png.cpp: Tool to create dataset with stereo pairs, ground-truth disparity and confidence
/dispflownet-adaptation-release/models/DispNet_Adaptation_kitti_train/: Contains solver and training procedure used to adapt DispNetCorrD1 on KITTI dataset + get_weights.sh to download a caffemodel weights file for DispNetCorr1D adpated on Kitti2012 using Census and CCNN for 120000 step
/dispflownet-adaptation-release/models/DispNet_Adaptation_ShadowOnTruck/: Contains get_weights.sh to download a caffemodel weights file for DispNetCorr1D adapted on the challenging sequence "ShadowOnTruck"

unsupervised-adaptation-for-deep-stereo's People

Contributors

Stargazers

Watchers

unsupervised-adaptation-for-deep-stereo's Issues

the link to tensorflow version "dispflownet-tf" is unsupervised?

hi, I prefer to use tensorflow, but it look like supervised version? because it need ground true when training?

thank you

The model is not convergent.

When I fine tuning the network with my dataset, The logs are as follows. I think the model is not convergent. The disparity computed by the fine-tuned model is a white image with a little texture.
Can you give me some suggestion about how to train the model? Or how to get a real reliable training dataset. Thanks.

I0508 15:53:59.874593  6385 solver.cpp:245]     Train net output #73719: blob65 = 74.7856
I0508 15:53:59.874598  6385 solver.cpp:245]     Train net output #73720: blob65 = 74.7856
I0508 15:53:59.874603  6385 solver.cpp:245]     Train net output #73721: blob65 = 74.7857
I0508 15:53:59.874608  6385 solver.cpp:245]     Train net output #73722: blob65 = 74.7857
I0508 15:53:59.874614  6385 solver.cpp:245]     Train net output #73723: blob65 = 74.7857
I0508 15:53:59.874619  6385 solver.cpp:245]     Train net output #73724: blob65 = 74.7856
I0508 15:53:59.874624  6385 solver.cpp:245]     Train net output #73725: blob65 = 74.7856
I0508 15:53:59.874629  6385 solver.cpp:245]     Train net output #73726: blob65 = 74.7856
I0508 15:53:59.874634  6385 solver.cpp:245]     Train net output #73727: blob65 = 74.7856
I0508 15:53:59.874646  6385 solver.cpp:245]     Train net output #73728: flow_loss1 = 0.0113061 (* 1 = 0.0113061 loss)
I0508 15:53:59.874652  6385 solver.cpp:245]     Train net output #73729: flow_loss2 = 174.297
I0508 15:53:59.874658  6385 solver.cpp:245]     Train net output #73730: flow_loss3 = 170.482
I0508 15:53:59.874665  6385 solver.cpp:245]     Train net output #73731: flow_loss4 = 163.814
I0508 15:53:59.874670  6385 solver.cpp:245]     Train net output #73732: flow_loss5 = 154.755
I0508 15:53:59.874675  6385 solver.cpp:245]     Train net output #73733: flow_loss6 = 141.625
I0508 15:53:59.874682  6385 solver.cpp:245]     Train net output #73734: loss_down = 0.00255099 (* 0.125 = 0.000318874 loss)
I0508 15:53:59.874689  6385 solver.cpp:245]     Train net output #73735: loss_est = 0.00268216 (* 0.125 = 0.00033527 loss)
I0508 15:53:59.874701  6385 solver.cpp:245]     Train net output #73736: loss_up = 0.00255099 (* 0.125 = 0.000318874 loss)
I0508 15:53:59.874707  6385 solver.cpp:245]     Train net output #73737: loss_west = 0.00268216 (* 0.125 = 0.00033527 loss)
I0508 15:53:59.874717  6385 sgd_solver.cpp:106] Iteration 11950, lr = 3.125e-06
I0508 15:55:18.302011  6385 solver.cpp:456] Snapshotting to binary proto file disp_iter_12000.caffemodel
I0508 15:55:18.302299  6385 net.cpp:867] Serializing 137 layers
I0508 15:55:21.718309  6385 sgd_solver.cpp:273] Snapshotting solver state to binary proto file disp_iter_12000.solverstate
I0508 15:55:29.855378  6385 solver.cpp:318] Iteration 12000, loss = 0.00952777
I0508 15:55:29.855406  6385 solver.cpp:323] Optimization Done.
I0508 15:55:29.855428  6385 caffe.cpp:222] Optimization Done.````

The source of training and test

In the train.prototxt, why the source for train and test are the same. Are they lmdb style?
source: "/media/Dataset/Stereo_Conf/KITTI12_CCNN"
layer {
name: "CustomData1"
type: "CustomData"
top: "blob0"
top: "blob1"
top: "blob2"
top: "blobConfidence"
include {
phase: TRAIN
}
data_param {
source: "/media/Dataset/Stereo_Conf/KITTI12_CCNN"
batch_size: 4
backend: LMDB
rand_permute: false
rand_permute_seed: 77
slice_point: 3
slice_point: 6
slice_point: 7
encoding: UINT8
encoding: UINT8
encoding: UINT16FLOW
encoding: CONF
verbose: true
}
}
layer {
name: "CustomData2"
type: "CustomData"
top: "blob0"
top: "blob1"
top: "blob2"
top: "blobConfidence"
include {
phase: TEST
}
data_param {
source: "/media/Dataset/Stereo_Conf/KITTI12_CCNN"
batch_size: 1
backend: LMDB
rand_permute: false
rand_permute_seed: 77
slice_point: 3
slice_point: 6
slice_point: 7
encoding: UINT8
encoding: UINT8
encoding: UINT16FLOW
encoding: CONF
verbose: true
}
}

src/caffe/layers/l1loss_layer.cpp:28:5: error: ‘diff_bottom_vec_’ was not declared in this scope

Hi @AlessioTonioni ,

After I replace the files in flownet with yours, something wrong while making

src/caffe/layers/l1loss_layer.cpp: In member function ‘virtual void caffe::L1LossLayer::LayerSetUp(const std::vector<caffe::Blob>&, const std::vector<caffe::Blob>&)’:
src/caffe/layers/l1loss_layer.cpp:28:5: error: ‘diff_bottom_vec_’ was not declared in this scope
diff_bottom_vec_.clear();
^
src/caffe/layers/l1loss_layer.cpp:43:7: error: ‘dot_top_vec_’ was not declared in this scope
dot_top_vec_.clear();
^
src/caffe/layers/l1loss_layer.cpp:44:31: error: ‘dot_’ was not declared in this scope
dot_top_vec_.push_back(&dot_);
^
src/caffe/layers/l1loss_layer.cpp:45:7: error: ‘dot_bottom_vec_’ was not declared in this scope
dot_bottom_vec_.clear();
^
src/caffe/layers/l1loss_layer.cpp:50:7: error: ‘dot_layer_’ was not declared in this scope
dot_layer_.reset(new EltwiseLayer(dot_param));
^
src/caffe/layers/l1loss_layer.cpp:53:7: error: ‘dot_top_vec_’ was not declared in this scope
dot_top_vec_=diff_top_vec_;
^
src/caffe/layers/l1loss_layer.cpp:70:26: error: ‘dot_top_vec_’ was not declared in this scope
square_layer_->SetUp(dot_top_vec_, square_top_vec_);
^
src/caffe/layers/l1loss_layer.cpp: In member function ‘virtual void caffe::L1LossLayer::Reshape(const std::vector<caffe::Blob>&, const std::vector<caffe::Blob>&)’:
src/caffe/layers/l1loss_layer.cpp:103:34: error: ‘dot_top_vec_’ was not declared in this scope
diff_layer_->Reshape(bottom, dot_top_vec_);
^
src/caffe/layers/l1loss_layer.cpp:115:28: error: ‘dot_top_vec_’ was not declared in this scope
square_layer_->Reshape(dot_top_vec_, square_top_vec_);
^
Makefile:576: recipe for target '.build_release/src/caffe/layers/l1loss_layer.o' failed
make: *** [.build_release/src/caffe/layers/l1loss_layer.o] Error 1

Pretrained model

Dear authors, first of all, thanks for the great work.
I'm wondering may you provide your pretrained model if possible?

Thank you

Inference results from tensorflow version

Hi, I tried with the tensorflow verision and ran the inference code with the pretrained model you provide. I run with the command line --fullRes.

The images of the disparity map look pretty well. But when I save the disparity as a numpy array directly, I realized compared with ground truth, it seems to have some scaling problems: It looks like that the disparity values have been divided by 2. I have no idea why it happens.

I have two examples to explain my question. One from Flyingthings and one from KITTI
https://drive.google.com/open?id=12ddoPxYLgetrO9KsmmLKzicPFA__uHhD
https://drive.google.com/open?id=1xW3IX5Ys225cxBsAY5fjTP1Q_ReyzvUE

input and filter must have the same depth: 3 vs 1

Dear authors, thanks for the great work.
I'm wondering may you provide the adcensus, SGM and CCNN code if possible?
To get the confidence map of disparity, I use the code from the author homepage https://github.com/fabiotosi92/CCNN-Tensorflow, it get perfect performance when test its own data. While I get the error when I use my disparity map by SGM and adcensus. The error is as follows:

tensorflow.python.framework.errors_impl.InvalidArgumentError: input and filter must have the same depth: 3 vs 1

[*] Testing....
 [*] Load model: SUCCESS
uint8uint8uint8uint8uint8hahahahahaha
 [*] Start Testing...
num_samples: [ 4]
 [*] Test image:./images/disparity/ad-census/81.png
hahahahahhahhaconfidenceconfidence
2018-04-24 09:11:43.337912: W tensorflow/core/kernels/queue_base.cc:294] _0_input_producer: Skipping cancelled enqueue attempt with queue not closed
Traceback (most recent call last):
  File "./model/main.py", line 47, in <module>
    tf.app.run()
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 48, in run
    _sys.exit(main(_sys.argv[:1] + flags_passthrough))
  File "./model/main.py", line 43, in main
    model.test(args)
  File "/media/jennifer/Papers/3_Codes_now/CCNN-Tensorflow/model/model.py", line 174, in test
    confidence = self.sess.run(png,feed_dict={self.disp: batch})
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 789, in run
    run_metadata_ptr)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 997, in _run
    feed_dict_string, options, run_metadata)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1132, in _do_run
    target_list, options, run_metadata)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1152, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: input and filter must have the same depth: 3 vs 1
	 [[Node: CCNN/conv1/conv/Conv2D = Conv2D[T=DT_FLOAT, data_format="NHWC", padding="VALID", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/gpu:0"](_arg_div_0_0/_71, CCNN/conv1/weights/read)]]

Caused by op u'CCNN/conv1/conv/Conv2D', defined at:
  File "./model/main.py", line 47, in <module>
    tf.app.run()
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 48, in run
    _sys.exit(main(_sys.argv[:1] + flags_passthrough))
  File "./model/main.py", line 37, in main
    model_name=args.model_name
  File "/media/jennifer/Papers/3_Codes_now/CCNN-Tensorflow/model/model.py", line 22, in __init__
    self.build_CCNN()
  File "/media/jennifer/Papers/3_Codes_now/CCNN-Tensorflow/model/model.py", line 45, in build_CCNN
    self.conv1 = ops.conv2d(self.disp, [kernel_size, kernel_size, 1, filters], 1, True, padding='VALID')
  File "/media/jennifer/Papers/3_Codes_now/CCNN-Tensorflow/model/ops.py", line 10, in conv2d
    x = tf.nn.conv2d(x, W, strides=[1, strides, strides, 1], padding=padding)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_nn_ops.py", line 399, in conv2d
    data_format=data_format, name=name)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py", line 767, in apply_op
    op_def=op_def)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2506, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1269, in __init__
    self._traceback = _extract_stack()

InvalidArgumentError (see above for traceback): input and filter must have the same depth: 3 vs 1
	 [[Node: CCNN/conv1/conv/Conv2D = Conv2D[T=DT_FLOAT, data_format="NHWC", padding="VALID", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/gpu:0"](_arg_div_0_0/_71, CCNN/conv1/weights/read)]]

Have you ever met the same error? hope for your help. Thanks.

Inference results from tensorflow version

Hi, I tried with the tensorflow verision and ran the inference code with the pretrained model you provide. I run with the command line --fullRes.

The images of the disparity map look pretty well. But when I save the disparity as a numpy array directly, I realized compared with ground truth, it seems to have some scaling problems.

How to increase the variety of training set

Dear sir, sorry to bother you again. In your paper, you perform random data augmentation (color, brightness and contrast transformantions) as done by the authors of DispNets. However, I still wonder how to do that. For example, there are four images: left image L and right image R, disparity map D obtained by sgm, confidence map C obtained by CCCN.
Should we directly perform random data augmentation on L, R , D and C ?
Or should we perform random data augmentation on L and R to get more left and right images L' and R'？We get more disparity map and confidence map based on L' and R'？

Hope for your reply. Thanks.

A question about paper, thanks your help.

Hi @AlessioTonioni, when I read your paper, I can't understand where you use unsupervised adaptation.
In your paper, SGM/AD-CENSUS + CCNN are used as ground truth labels(in your paper you write 'CCCN'), but when i read the CCNN papar, it just use the kitti's ground truth to calculate the confidence, so can you tell me how to deal with datasets that don't have ground truth?
Thx.

Check failed: MaxBottomBlobs() >= bottom.size() (2 vs. 3) L1Loss Layer takes at most 2 bottom blob(s) as input.

Dear sir, sorry to bother you. I have meet an error when I fine-tune your code with my dataset. The error is:
F0427 17:12:09.233387 607 layer.hpp:404] Check failed: MaxBottomBlobs() >= bottom.size() (2 vs. 3) L1Loss Layer takes at most 2 bottom blob(s) as input.

I0427 17:12:09.233264   607 net.cpp:149] Top shape: 4 1 192 384 (294912)
I0427 17:12:09.233271   607 net.cpp:149] Top shape: 4 1 192 384 (294912)
I0427 17:12:09.233276   607 net.cpp:149] Top shape: 4 1 192 384 (294912)
I0427 17:12:09.233283   607 net.cpp:149] Top shape: 4 1 192 384 (294912)
I0427 17:12:09.233288   607 net.cpp:149] Top shape: 4 1 192 384 (294912)
I0427 17:12:09.233291   607 net.cpp:157] Memory required for data: 2190802964
I0427 17:12:09.233299   607 layer_factory.hpp:77] Creating layer flow_loss1
I0427 17:12:09.233316   607 net.cpp:91] Creating Layer flow_loss1
I0427 17:12:09.233325   607 net.cpp:426] flow_loss1 <- blob66_NegReLU6_0_split_0
I0427 17:12:09.233337   607 net.cpp:426] flow_loss1 <- blob65
I0427 17:12:09.233346   607 net.cpp:426] flow_loss1 <- blob65Confidence
I0427 17:12:09.233358   607 net.cpp:400] flow_loss1 -> flow_loss1
F0427 17:12:09.233387   607 layer.hpp:404] Check failed: MaxBottomBlobs() >= bottom.size() (2 vs. 3) L1Loss Layer takes at most 2 bottom blob(s) as input.
*** Check failure stack trace: ***
    @     0x7fd1a56dd5cd  google::LogMessage::Fail()
    @     0x7fd1a56df433  google::LogMessage::SendToLog()
    @     0x7fd1a56dd15b  google::LogMessage::Flush()
    @     0x7fd1a56dfe1e  google::LogMessageFatal::~LogMessageFatal()
    @     0x7fd1a5f92ff7  caffe::Layer<>::CheckBlobCounts()
    @     0x7fd1a5f92113  caffe::Layer<>::SetUp()
    @     0x7fd1a60aea08  caffe::Net<>::Init()
    @     0x7fd1a60acd51  caffe::Net<>::Net()
    @     0x7fd1a60e0101  caffe::Solver<>::InitTrainNet()
    @     0x7fd1a60df8ea  caffe::Solver<>::Init()
    @     0x7fd1a60df334  caffe::Solver<>::Solver()
    @     0x7fd1a60ed31b  caffe::SGDSolver<>::SGDSolver()
    @     0x7fd1a60f1e04  caffe::AdamSolver<>::AdamSolver()
    @     0x7fd1a60f2d38  caffe::Creator_AdamSolver<>()
    @           0x420d66  caffe::SolverRegistry<>::CreateSolver()
    @           0x41c0c4  train()
    @           0x41e639  main
    @     0x7fd1a4191830  __libc_start_main
    @           0x41ae39  _start
    @              (nil)  (unknown)
已放弃 (核心已转储)

Have you ever met this error again? Thanks very much.

cvlab-unibo / unsupervised-adaptation-for-deep-stereo Goto Github PK

unsupervised-adaptation-for-deep-stereo's Introduction

Unsupervised-Adaptation-for-Deep-Stereo

List of source files:

unsupervised-adaptation-for-deep-stereo's People

Contributors

Stargazers

Watchers

Forkers

unsupervised-adaptation-for-deep-stereo's Issues

Recommend Projects

Recommend Topics

Recommend Org