timosaemann / enet Goto Github PK

Hi I am following your tutorial to train on the Cityscape dataset.

I am running the following code for the ENet predictions.

python calculate_class_weighting.py --source ENet/dataset/train_fine_cityscapes.txt --num_classes 19 

but i get:

Traceback (most recent call last):
  File "calculate_class_weighting.py", line 48, in <module>
    if ((labels == i).sum()) == 0:
AttributeError: 'bool' object has no attribute 'sum'

thank you

Hi Timo, have you ever trained ENet on indoor dataset, such as SUNRGBD? If so, could you please offer your trained caffemodel? Thanks.

hi, I use the calculate_class_weighting.py generare the loss weight for my training.
and I find some problems.

First, the weight is：
0.0819
0.4754
0.1324
1.5224
1.5190
2.4730
8.1865
5.2286
0.1870
1.4695
0.6893
1.9814
7.8091
0.4164
1.3809
1.1982
0.6273
5.3535
4.0939
,the distrubution is too large.

Second, I use the weight for loss to train, And the Mean IOU drop 7%.
Could you give me some tricks and help?
thx a lot!

Hi I was just wondering if you can publicise the accuracy of the model trained on your setup (in term of class IOU and category IOU). I was trying out your caffe implementation but seem like the results I got were not as good as what were described in the paper.

Is it possible to convert a pre-trained ENet into a half-precision version?
Thanks

Thanks for this useful code. I'm trying to finetune the encoder-decoder model using cityscapes_weights.caffemodel. My own data is 512x512 RGB with 2 classes, so I think maybe this size discrepancy is causing the following error (after initializing from prototxt):

I0707 13:02:04.584967 59686 net.cpp:283] Network initialization done.
I0707 13:02:04.586356 59686 solver.cpp:60] Solver scaffolding done.
I0707 13:02:04.612700 59686 caffe.cpp:155] Finetuning from /media/borges/pl/enet/weights/cityscapes_weights.caffemodel
F0707 13:02:04.719262 59686 net.cpp:767] Check failed: target_blobs.size() == source_layer.blobs_size() (1 vs. 2) Incompatible number of blobs for layer conv1_0_0

Do you have any insight into what might be causing this? I'm using all of the default settings other than commenting the resize params in the input blob, changing the number of classes in the deconvolution layer from 19 to 2, and adding class frequency weighting in the softmax layer.

hi, when I running the step
ENet/caffe-enet/build/tools/caffe train -solver /ENet/prototxts/enet_solver_encoder.prototxt
get the following error:
I1112 21:57:49.637869 27309 layer_factory.hpp:77] Creating layer drop1_0_3
[libprotobuf ERROR google/protobuf/descriptor_database.cc:58] File already exists in database: google/protobuf/descriptor.proto
[libprotobuf FATAL google/protobuf/descriptor.cc:1394] CHECK failed: generated_database_->Add(encoded_file_descriptor, size):
terminate called after throwing an instance of 'google::protobuf::FatalException'
what(): CHECK failed: generated_database_->Add(encoded_file_descriptor, size)

any suggest?
thanks

The path the project ENet /home/user/lib/caffe/ENet/

dataset is
train_fine_cityscapes.txt

enet_solver_encoder.prototxt:
net: "/home/user/lib/caffe/ENet/prototxts/enet_train_encoder.prototxt" # Change this to the absolute path to your model file
test_initialization: true
test_iter: 1
test_interval: 10000000
base_lr: 0.005
lr_policy: "multistep"
gamma: 0.1
stepvalue: 18750
stepvalue: 37500
stepvalue: 56250
display: 20
momentum: 0.9
max_iter: 75000
weight_decay: 0.0002
snapshot: 10000
snapshot_prefix: "/home/user/lib/caffe/ENet/weights/snapshots_encoder/enet" # Change this to the absolute path to where you wish to output solver snapshots
solver_mode: GPU
solver_type: ADAM
momentum2: 0.999

When I call the command /home/user/lib/caffe/ENet/caffe-enet/build/tools/caffe train -solver /home/user/lib/caffe/ENet/prototxts/enet_solver_encoder.prototxt

I get this result:
...

I0728 12:07:31.305349 32224 layer_factory.hpp:77] Creating layer bn1_0_2
I0728 12:07:31.305358 32224 net.cpp:100] Creating Layer bn1_0_2
I0728 12:07:31.305361 32224 net.cpp:434] bn1_0_2 <- conv1_0_2
I0728 12:07:31.305366 32224 net.cpp:408] bn1_0_2 -> bn1_0_2
I0728 12:07:31.305560 32224 net.cpp:150] Setting up bn1_0_2
I0728 12:07:31.305565 32224 net.cpp:157] Top shape: 4 64 128 256 (8388608)
I0728 12:07:31.305567 32224 net.cpp:165] Memory required for data: 394657792
I0728 12:07:31.305574 32224 layer_factory.hpp:77] Creating layer drop1_0_3
*** Aborted at 1501232851 (unix time) try "date -d @1501232851" if you are using GNU date ***
PC: @ 0x7f9a8456db73 std::_Hashtable<>::clear()
*** SIGSEGV (@0x9) received by PID 32224 (TID 0x7f9aead65ac0) from PID 9; stack trace: ***
@ 0x7f9ae83b84b0 (unknown)
@ 0x7f9a8456db73 std::_Hashtable<>::clear()
@ 0x7f9a8455eca6 google::protobuf::DescriptorPool::FindFileByName()
@ 0x7f9a8453bdc8 google::protobuf::python::cdescriptor_pool::AddSerializedFile()
@ 0x7f9ae8ff07d0 PyEval_EvalFrameEx
@ 0x7f9ae911901c PyEval_EvalCodeEx
@ 0x7f9ae906f3dd (unknown)
@ 0x7f9ae90421e3 PyObject_Call
@ 0x7f9ae9062ae5 (unknown)
@ 0x7f9ae8ff9123 (unknown)
@ 0x7f9ae90421e3 PyObject_Call
@ 0x7f9ae8fed13c PyEval_EvalFrameEx
@ 0x7f9ae911901c PyEval_EvalCodeEx
@ 0x7f9ae8fe7b89 PyEval_EvalCode
@ 0x7f9ae907c1b4 PyImport_ExecCodeModuleEx
@ 0x7f9ae907cb8f (unknown)
@ 0x7f9ae907e300 (unknown)
@ 0x7f9ae907e5c8 (unknown)
@ 0x7f9ae907f6db PyImport_ImportModuleLevel
@ 0x7f9ae8ff6698 (unknown)
@ 0x7f9ae90421e3 PyObject_Call
@ 0x7f9ae9118447 PyEval_CallObjectWithKeywords
@ 0x7f9ae8feb5c6 PyEval_EvalFrameEx
@ 0x7f9ae911901c PyEval_EvalCodeEx
@ 0x7f9ae8fe7b89 PyEval_EvalCode
@ 0x7f9ae907c1b4 PyImport_ExecCodeModuleEx
@ 0x7f9ae907cb8f (unknown)
@ 0x7f9ae907e300 (unknown)
@ 0x7f9ae907e5c8 (unknown)
@ 0x7f9ae907f6db PyImport_ImportModuleLevel
@ 0x7f9ae8ff6698 (unknown)
@ 0x7f9ae90421e3 PyObject_Call

Could you help?

I am facing the following issue while building the cmake file

-- Boost version: 1.53.0
-- Found the following Boost libraries:
-- system
-- thread
-- filesystem
-- Could NOT find GFlags (missing: GFLAGS_INCLUDE_DIR GFLAGS_LIBRARY)
-- Could NOT find Glog (missing: GLOG_INCLUDE_DIR GLOG_LIBRARY)
-- Could NOT find PROTOBUF (missing: PROTOBUF_LIBRARY PROTOBUF_INCLUDE_DIR)
-- Found PROTOBUF Compiler: /home/tpi/anaconda3/bin/protoc
CMake Error at /usr/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:108 (message):
Could NOT find HDF5 (missing: HDF5_LIBRARIES HDF5_INCLUDE_DIRS)
Call Stack (most recent call first):
/usr/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:315 (_FPHSA_FAILURE_MESSAGE)
/usr/share/cmake/Modules/FindHDF5.cmake:381 (find_package_handle_standard_args)
cmake/Dependencies.cmake:27 (find_package)
CMakeLists.txt:43 (include)

CMake Error: The following variables are used in this project, but they are set to NOTFOUND.
Please set them or make sure they are set and tested correctly in the CMake files:
PROTOBUF_INCLUDE_DIR (ADVANCED)
used as include directory in directory /home/tpi/asad/ENet/caffe-enet
used as include directory in directory /home/tpi/asad/ENet/caffe-enet
used as include directory in directory /home/tpi/asad/ENet/caffe-enet

I1008 16:19:57.421929 26865 net.cpp:165] Memory required for data: 285605888
I1008 16:19:57.421936 26865 layer_factory.hpp:77] Creating layer bn1_0_0
I1008 16:19:57.421943 26865 net.cpp:100] Creating Layer bn1_0_0
I1008 16:19:57.421947 26865 net.cpp:434] bn1_0_0 <- conv1_0_0
I1008 16:19:57.421952 26865 net.cpp:408] bn1_0_0 -> bn1_0_0
I1008 16:19:57.422133 26865 net.cpp:150] Setting up bn1_0_0
I1008 16:19:57.422152 26865 net.cpp:157] Top shape: 4 16 128 256 (2097152)
I1008 16:19:57.422157 26865 net.cpp:165] Memory required for data: 293994496
I1008 16:19:57.422164 26865 layer_factory.hpp:77] Creating layer prelu1_0_0
I1008 16:19:57.422173 26865 net.cpp:100] Creating Layer prelu1_0_0
I1008 16:19:57.422175 26865 net.cpp:434] prelu1_0_0 <- bn1_0_0
I1008 16:19:57.422181 26865 net.cpp:408] prelu1_0_0 -> prelu1_0_0
I1008 16:19:57.422938 26865 net.cpp:150] Setting up prelu1_0_0
I1008 16:19:57.422947 26865 net.cpp:157] Top shape: 4 16 128 256 (2097152)
I1008 16:19:57.422968 26865 net.cpp:165] Memory required for data: 302383104
I1008 16:19:57.422976 26865 layer_factory.hpp:77] Creating layer conv1_0_1
I1008 16:19:57.422982 26865 net.cpp:100] Creating Layer conv1_0_1
I1008 16:19:57.422986 26865 net.cpp:434] conv1_0_1 <- prelu1_0_0
I1008 16:19:57.422991 26865 net.cpp:408] conv1_0_1 -> conv1_0_1
I1008 16:19:57.424350 26865 net.cpp:150] Setting up conv1_0_1
I1008 16:19:57.424381 26865 net.cpp:157] Top shape: 4 16 128 256 (2097152)
I1008 16:19:57.424382 26865 net.cpp:165] Memory required for data: 310771712
I1008 16:19:57.424389 26865 layer_factory.hpp:77] Creating layer bn1_0_1
I1008 16:19:57.424397 26865 net.cpp:100] Creating Layer bn1_0_1
I1008 16:19:57.424401 26865 net.cpp:434] bn1_0_1 <- conv1_0_1
I1008 16:19:57.424418 26865 net.cpp:408] bn1_0_1 -> bn1_0_1
I1008 16:19:57.424620 26865 net.cpp:150] Setting up bn1_0_1
I1008 16:19:57.424625 26865 net.cpp:157] Top shape: 4 16 128 256 (2097152)
I1008 16:19:57.424645 26865 net.cpp:165] Memory required for data: 319160320
I1008 16:19:57.424649 26865 layer_factory.hpp:77] Creating layer prelu1_0_1
I1008 16:19:57.424654 26865 net.cpp:100] Creating Layer prelu1_0_1
I1008 16:19:57.424656 26865 net.cpp:434] prelu1_0_1 <- bn1_0_1
I1008 16:19:57.424660 26865 net.cpp:408] prelu1_0_1 -> prelu1_0_1
I1008 16:19:57.425477 26865 net.cpp:150] Setting up prelu1_0_1
I1008 16:19:57.425487 26865 net.cpp:157] Top shape: 4 16 128 256 (2097152)
I1008 16:19:57.425489 26865 net.cpp:165] Memory required for data: 327548928
I1008 16:19:57.425493 26865 layer_factory.hpp:77] Creating layer conv1_0_2
I1008 16:19:57.425501 26865 net.cpp:100] Creating Layer conv1_0_2
I1008 16:19:57.425504 26865 net.cpp:434] conv1_0_2 <- prelu1_0_1
I1008 16:19:57.425509 26865 net.cpp:408] conv1_0_2 -> conv1_0_2
I1008 16:19:57.426268 26865 net.cpp:150] Setting up conv1_0_2
I1008 16:19:57.426277 26865 net.cpp:157] Top shape: 4 64 128 256 (8388608)
I1008 16:19:57.426280 26865 net.cpp:165] Memory required for data: 361103360
I1008 16:19:57.426285 26865 layer_factory.hpp:77] Creating layer bn1_0_2
I1008 16:19:57.426319 26865 net.cpp:100] Creating Layer bn1_0_2
I1008 16:19:57.426323 26865 net.cpp:434] bn1_0_2 <- conv1_0_2
I1008 16:19:57.426327 26865 net.cpp:408] bn1_0_2 -> bn1_0_2
I1008 16:19:57.426498 26865 net.cpp:150] Setting up bn1_0_2
I1008 16:19:57.426502 26865 net.cpp:157] Top shape: 4 64 128 256 (8388608)
I1008 16:19:57.426506 26865 net.cpp:165] Memory required for data: 394657792
I1008 16:19:57.426512 26865 layer_factory.hpp:77] Creating layer drop1_0_3
*** Aborted at 1507504798 (unix time) try "date -d @1507504798" if you are using GNU date ***
PC: @ 0x7fcff5b417a3 std::_Hashtable<>::clear()
*** SIGSEGV (@0x9) received by PID 26865 (TID 0x7fd0429d0ac0) from PID 9; stack trace: ***
@ 0x7fd03fffc4b0 (unknown)
@ 0x7fcff5b417a3 std::_Hashtable<>::clear()
@ 0x7fcff5b3a116 google::protobuf::DescriptorPool::FindFileByName()
@ 0x7fcff5af3f98 google::protobuf::python::cdescriptor_pool::AddSerializedFile()
@ 0x7fd040c347d0 PyEval_EvalFrameEx
@ 0x7fd040d5d01c PyEval_EvalCodeEx
@ 0x7fd040cb33dd (unknown)
@ 0x7fd040c861e3 PyObject_Call
@ 0x7fd040ca6ae5 (unknown)
@ 0x7fd040c3d123 (unknown)
@ 0x7fd040c861e3 PyObject_Call
@ 0x7fd040c3113c PyEval_EvalFrameEx
@ 0x7fd040d5d01c PyEval_EvalCodeEx
@ 0x7fd040c2bb89 PyEval_EvalCode
@ 0x7fd040cc01b4 PyImport_ExecCodeModuleEx
@ 0x7fd040cc0b8f (unknown)
@ 0x7fd040cc2300 (unknown)
@ 0x7fd040cc25c8 (unknown)
@ 0x7fd040cc36db PyImport_ImportModuleLevel
@ 0x7fd040c3a698 (unknown)
@ 0x7fd040c861e3 PyObject_Call
@ 0x7fd040d5c447 PyEval_CallObjectWithKeywords
@ 0x7fd040c2f5c6 PyEval_EvalFrameEx
@ 0x7fd040d5d01c PyEval_EvalCodeEx
@ 0x7fd040c2bb89 PyEval_EvalCode
@ 0x7fd040cc01b4 PyImport_ExecCodeModuleEx
@ 0x7fd040cc0b8f (unknown)
@ 0x7fd040cc2300 (unknown)
@ 0x7fd040cc25c8 (unknown)
@ 0x7fd040cc36db PyImport_ImportModuleLevel
@ 0x7fd040c3a698 (unknown)
@ 0x7fd040c861e3 PyObject_Call
Segmentation fault (core dumped)

any idea?

Thank you,
-Victor

I am receiving this error when i try to run the pre-trained model

WARNING: Logging before InitGoogleLogging() is written to STDERR
W0713 03:50:08.101807 13390 _caffe.cpp:135] DEPRECATION WARNING - deprecated use of Python interface
W0713 03:50:08.101938 13390 _caffe.cpp:136] Use this instead (with the named "weights" parameter):
W0713 03:50:08.101943 13390 _caffe.cpp:138] Net('ENet/prototxts/enet_deploy_final.prototxt', 1, weights='ENet/enet_weights_zoo/cityscapes_weights.caffemodel')
Traceback (most recent call last):
File "test_segmentation.py", line 46, in
net = caffe.Net(args.model, args.weights, caffe.TEST)
RuntimeError: Could not open file ENet/prototxts/enet_deploy_final.prototxt

I want to train the encoder model but I encounter the following error:
F0721 17:34:26.930198 1506 math_functions.cu:121] Check failed: status == CUBLAS_STATUS_SUCCESS (11 vs. 0) CUBLAS_STATUS_MAPPING_ERROR

The part of the prototxt is shown as below:
layer {
name: "loss"
type: "SoftmaxWithLoss"
bottom: "deconv_encoder6_0_0"
bottom: "label"
top: "loss"
loss_param {
ignore_label: 255
weight_by_label_freqs: true
class_weighting: 64.0873
class_weighting: 0.5648
class_weighting: 0.7328
class_weighting: 1.7165
class_weighting: 1.9094
class_weighting: 4.1597
class_weighting: 0.7704
class_weighting: 0.0782
class_weighting: 0.4542
class_weighting: 1.0008
class_weighting: 0.4880
class_weighting: 0.1264
class_weighting: 1.4550
class_weighting: 1.4524
class_weighting: 1.8854
class_weighting: 0.7084
class_weighting: 0.4037
class_weighting: 2.3622
class_weighting: 24.2244
}
}
layer {
name: "accuracy"
type: "Accuracy"
bottom: "deconv_encoder6_0_0"
bottom: "label"
top: "accuracy"
top: "per_class_accuracy"
}
The the GT of cityscapes dataset is downloaded from the official website. And it is not created by the propvided scripts. To be specific, the background is 0 not 255.
I guess the error is the labelId mapping problem. How to solve it?
Thanks!

The tutorial says that after running bn absorber the BN layers should be removed in final prototxt files, however this script removed only dropout layers and changed BN layers parameters, but did not remove them. BN layers are also not removed in https://github.com/TimoSaemann/ENet/blob/master/prototxts/enet_deploy_final.prototxt

Edit: Apparently only BN layers following conv layers being merged. The rest 3 BN layers make no big difference, this is why closing the issue.

Hi,

I am training the caffe version of ENet. When training on the cityscape fine annotated dataset, my training accuracy seems not to be converged. Do you have any suggestion about how to solve it?
All parameters are the default one except for the batch is set as 2 due to the memory capacity.

Thanks for your help.

I1203 11:35:57.409646 12188 solver.cpp:228] Iteration 173240, loss = 1.04035
I1203 11:35:57.409780 12188 solver.cpp:244] Train net output #0: accuracy = 0.609447
I1203 11:35:57.409791 12188 solver.cpp:244] Train net output #1: loss = 1.04036 (* 1 = 1.04036 loss)
I1203 11:35:57.409796 12188 solver.cpp:244] Train net output #2: per_class_accuracy = 0.799779
I1203 11:35:57.409801 12188 solver.cpp:244] Train net output #3: per_class_accuracy = 0.729202
I1203 11:35:57.409806 12188 solver.cpp:244] Train net output #4: per_class_accuracy = 0.754431
I1203 11:35:57.409809 12188 solver.cpp:244] Train net output #5: per_class_accuracy = 0
I1203 11:35:57.409813 12188 solver.cpp:244] Train net output #6: per_class_accuracy = 0.298023
I1203 11:35:57.409817 12188 solver.cpp:244] Train net output #7: per_class_accuracy = 0.747144
I1203 11:35:57.409821 12188 solver.cpp:244] Train net output #8: per_class_accuracy = 0.841352
I1203 11:35:57.409826 12188 solver.cpp:244] Train net output #9: per_class_accuracy = 0.406868
I1203 11:35:57.409828 12188 solver.cpp:244] Train net output #10: per_class_accuracy = 0.627141
I1203 11:35:57.409832 12188 solver.cpp:244] Train net output #11: per_class_accuracy = 0.809035
I1203 11:35:57.409837 12188 solver.cpp:244] Train net output #12: per_class_accuracy = 0.98963
I1203 11:35:57.409842 12188 solver.cpp:244] Train net output #13: per_class_accuracy = 0.215297
I1203 11:35:57.409845 12188 solver.cpp:244] Train net output #14: per_class_accuracy = 0
I1203 11:35:57.409849 12188 solver.cpp:244] Train net output #15: per_class_accuracy = 0.578774
I1203 11:35:57.409853 12188 solver.cpp:244] Train net output #16: per_class_accuracy = 0
I1203 11:35:57.409857 12188 solver.cpp:244] Train net output #17: per_class_accuracy = 0
I1203 11:35:57.409862 12188 solver.cpp:244] Train net output #18: per_class_accuracy = 0
I1203 11:35:57.409865 12188 solver.cpp:244] Train net output #19: per_class_accuracy = 0
I1203 11:35:57.409869 12188 solver.cpp:244] Train net output #20: per_class_accuracy = 0

Hi, when executing sh enet_weights_zoo/cityscapes_weights.sh it gives the error message ... unzip: cannot find zipfile directory in one of /ENet/enet_weights_zoo/weights2.caffemodel ... The file does not seem to be a zip file.

During training, the BN layer will output NAN value. It's probably due to the base CAFFE version, please refer to the following link for the details:
BVLC/caffe#5136

I am trying to train the network with a custom dataset in DIGITs but I am getting the error "Unsupported data layer type DenseImageData".

Traceback (most recent call last):
  File "digits/scheduler.py", line 512, in run_task
    task.run(resources)
  File "digits/task.py", line 189, in run
    self.before_run()
  File "digits/model/tasks/caffe_train.py", line 220, in before_run
    self.save_files_generic()
  File "digits/model/tasks/caffe_train.py", line 632, in save_files_generic
    network = cleanedUpGenericNetwork(self.network)
  File "digits/model/tasks/caffe_train.py", line 1701, in cleanedUpGenericNetwork
    'Unsupported data layer type %s' % layer.type
AssertionError: Unsupported data layer type DenseImageData

I looked for it in the caffe-enet submodule and it is supposed to be registered on there in the cpp code. But it fails when I try to train the network

Thanks in advance.

Hi @TimoSaemann,

I would like to know what label_divide_factor means in the prototxt files? What it is used for ? Why is it set to 8 when we generate the prototxt file?
Could you please explain it to me?

Thank you very much!

Hello,
I have a question.

When I train, I met the error.

I0626 10:41:47.785637 4806 layer_factory.hpp:77] Creating layer data
I0626 10:41:47.786315 4806 net.cpp:100] Creating Layer data
I0626 10:41:47.786324 4806 net.cpp:408] data -> data
I0626 10:41:47.786337 4806 net.cpp:408] data -> label
I0626 10:41:47.786352 4806 dense_image_data_layer.cpp:50] Opening file dataset/train_fine_cityscapes.txt
I0626 10:41:47.788602 4806 dense_image_data_layer.cpp:60] Shuffling data
I0626 10:41:47.789371 4806 dense_image_data_layer.cpp:65] A total of 2975 examples.
I0626 10:41:47.857427 4806 dense_image_data_layer.cpp:118] output data size top[0]: 3,3,512,1024
I0626 10:41:47.857450 4806 dense_image_data_layer.cpp:121] output data size top[1]: 3,1,64,128
I0626 10:41:47.862567 4806 net.cpp:150] Setting up data
I0626 10:41:47.862602 4806 net.cpp:157] Top shape: 3 3 512 1024 (4718592)
I0626 10:41:47.862608 4806 net.cpp:157] Top shape: 3 1 64 128 (24576)
I0626 10:41:47.862612 4806 net.cpp:165] Memory required for data: 18972672
I0626 10:41:47.862619 4806 layer_factory.hpp:77] Creating layer data_data_0_split
I0626 10:41:47.862988 4806 net.cpp:100] Creating Layer data_data_0_split
I0626 10:41:47.862996 4806 net.cpp:434] data_data_0_split <- data
I0626 10:41:47.863008 4806 net.cpp:408] data_data_0_split -> data_data_0_split_0
I0626 10:41:47.863015 4806 net.cpp:408] data_data_0_split -> data_data_0_split_1
I0626 10:41:47.863376 4806 net.cpp:150] Setting up data_data_0_split
I0626 10:41:47.863386 4806 net.cpp:157] Top shape: 3 3 512 1024 (4718592)
I0626 10:41:47.863391 4806 net.cpp:157] Top shape: 3 3 512 1024 (4718592)
I0626 10:41:47.863394 4806 net.cpp:165] Memory required for data: 56721408
I0626 10:41:47.863415 4806 layer_factory.hpp:77] Creating layer label_data_1_split
I0626 10:41:47.863423 4806 net.cpp:100] Creating Layer label_data_1_split
I0626 10:41:47.863430 4806 net.cpp:434] label_data_1_split <- label
I0626 10:41:47.863436 4806 net.cpp:408] label_data_1_split -> label_data_1_split_0
I0626 10:41:47.863445 4806 net.cpp:408] label_data_1_split -> label_data_1_split_1
I0626 10:41:47.863474 4806 net.cpp:150] Setting up label_data_1_split
I0626 10:41:47.863483 4806 net.cpp:157] Top shape: 3 1 64 128 (24576)
I0626 10:41:47.863488 4806 net.cpp:157] Top shape: 3 1 64 128 (24576)
I0626 10:41:47.863493 4806 net.cpp:165] Memory required for data: 56918016
I0626 10:41:47.863497 4806 layer_factory.hpp:77] Creating layer conv0_1
I0626 10:41:47.863513 4806 net.cpp:100] Creating Layer conv0_1
I0626 10:41:47.863519 4806 net.cpp:434] conv0_1 <- data_data_0_split_0
I0626 10:41:47.863526 4806 net.cpp:408] conv0_1 -> conv0_1
F0626 10:41:48.349544 4806 cudnn.hpp:113] Check failed: status == CUDNN_STATUS_SUCCESS (3 vs. 0) CUDNN_STATUS_BAD_PARAM
*** Check failure stack trace: ***
@ 0x7f0005e46daa (unknown)
@ 0x7f0005e46ce4 (unknown)
@ 0x7f0005e466e6 (unknown)
@ 0x7f0005e49687 (unknown)
@ 0x7f00064a3dc6 caffe::CuDNNConvolutionLayer<>::Reshape()
@ 0x7f0006433045 caffe::Net<>::Init()
@ 0x7f0006433ed5 caffe::Net<>::Net()
@ 0x7f00063cccea caffe::Solver<>::InitTrainNet()
@ 0x7f00063cdefc caffe::Solver<>::Init()
@ 0x7f00063ce22a caffe::Solver<>::Solver()
@ 0x7f00063d7b33 caffe::Creator_AdamSolver<>()
@ 0x411ff6 caffe::SolverRegistry<>::CreateSolver()
@ 0x40af72 train()
@ 0x4089ac main
@ 0x7f000441af45 (unknown)
@ 0x4092b3 (unknown)
@ (nil) (unknown)
Aborted (core dumped)

my cudnn version is 5.1.0

Hi. I'm trying to reproduce the results with class weights.

Would you please say have you trained the ENet with class weights successfully?

This happens every time(at different iterations) I try to train the decoder with class weights:

I0720 02:37:59.913910 13471 solver.cpp:228] Iteration 3140, loss = 0.604358
I0720 02:37:59.913946 13471 solver.cpp:244]     Train net output #0: accuracy = 0.490742
I0720 02:37:59.913954 13471 solver.cpp:244]     Train net output #1: loss = 0.604358 (* 1 = 0.604358 loss)
I0720 02:37:59.913959 13471 solver.cpp:244]     Train net output #2: per_class_accuracy = 0.582444
I0720 02:37:59.913961 13471 solver.cpp:244]     Train net output #3: per_class_accuracy = 0.479704
I0720 02:37:59.913964 13471 solver.cpp:244]     Train net output #4: per_class_accuracy = 0.330785
I0720 02:37:59.913969 13471 solver.cpp:244]     Train net output #5: per_class_accuracy = 0
I0720 02:37:59.913971 13471 solver.cpp:244]     Train net output #6: per_class_accuracy = 0.332091
I0720 02:37:59.913975 13471 solver.cpp:244]     Train net output #7: per_class_accuracy = 0.487523
I0720 02:37:59.913977 13471 solver.cpp:244]     Train net output #8: per_class_accuracy = 0.552928
I0720 02:37:59.913981 13471 solver.cpp:244]     Train net output #9: per_class_accuracy = 0.650392
I0720 02:37:59.913985 13471 solver.cpp:244]     Train net output #10: per_class_accuracy = 0.562122
I0720 02:37:59.913987 13471 solver.cpp:244]     Train net output #11: per_class_accuracy = 0
I0720 02:37:59.913991 13471 solver.cpp:244]     Train net output #12: per_class_accuracy = 0.976575
I0720 02:37:59.913995 13471 solver.cpp:244]     Train net output #13: per_class_accuracy = 0.738946
I0720 02:37:59.913997 13471 solver.cpp:244]     Train net output #14: per_class_accuracy = 0
I0720 02:37:59.914001 13471 solver.cpp:244]     Train net output #15: per_class_accuracy = 0.658808
I0720 02:37:59.914005 13471 solver.cpp:244]     Train net output #16: per_class_accuracy = 0
I0720 02:37:59.914011 13471 solver.cpp:244]     Train net output #17: per_class_accuracy = 0
I0720 02:37:59.914014 13471 solver.cpp:244]     Train net output #18: per_class_accuracy = 0
I0720 02:37:59.914034 13471 solver.cpp:244]     Train net output #19: per_class_accuracy = 0
I0720 02:37:59.914049 13471 solver.cpp:244]     Train net output #20: per_class_accuracy = 0
I0720 02:37:59.914054 13471 sgd_solver.cpp:106] Iteration 3140, lr = 0.005
I0720 02:38:23.496275 13471 solver.cpp:228] Iteration 3160, loss = 25.5314
I0720 02:38:23.496443 13471 solver.cpp:244]     Train net output #0: accuracy = 0.00153637
I0720 02:38:23.496454 13471 solver.cpp:244]     Train net output #1: loss = 25.5314 (* 1 = 25.5314 loss)
I0720 02:38:23.496459 13471 solver.cpp:244]     Train net output #2: per_class_accuracy = 0
I0720 02:38:23.496461 13471 solver.cpp:244]     Train net output #3: per_class_accuracy = 0
I0720 02:38:23.496464 13471 solver.cpp:244]     Train net output #4: per_class_accuracy = 0
I0720 02:38:23.496467 13471 solver.cpp:244]     Train net output #5: per_class_accuracy = 0
I0720 02:38:23.496471 13471 solver.cpp:244]     Train net output #6: per_class_accuracy = 0
I0720 02:38:23.496474 13471 solver.cpp:244]     Train net output #7: per_class_accuracy = 0
I0720 02:38:23.496477 13471 solver.cpp:244]     Train net output #8: per_class_accuracy = 0
I0720 02:38:23.496480 13471 solver.cpp:244]     Train net output #9: per_class_accuracy = 0
I0720 02:38:23.496484 13471 solver.cpp:244]     Train net output #10: per_class_accuracy = 0
I0720 02:38:23.496487 13471 solver.cpp:244]     Train net output #11: per_class_accuracy = 0
I0720 02:38:23.496490 13471 solver.cpp:244]     Train net output #12: per_class_accuracy = 0
I0720 02:38:23.496493 13471 solver.cpp:244]     Train net output #13: per_class_accuracy = 0
I0720 02:38:23.496496 13471 solver.cpp:244]     Train net output #14: per_class_accuracy = 0
I0720 02:38:23.496500 13471 solver.cpp:244]     Train net output #15: per_class_accuracy = 0
I0720 02:38:23.496505 13471 solver.cpp:244]     Train net output #16: per_class_accuracy = 0
I0720 02:38:23.496510 13471 solver.cpp:244]     Train net output #17: per_class_accuracy = 0
I0720 02:38:23.496512 13471 solver.cpp:244]     Train net output #18: per_class_accuracy = 0
I0720 02:38:23.496516 13471 solver.cpp:244]     Train net output #19: per_class_accuracy = 0
I0720 02:38:23.496520 13471 solver.cpp:244]     Train net output #20: per_class_accuracy = 1
I0720 02:38:23.496525 13471 sgd_solver.cpp:106] Iteration 3160, lr = 0.005

After removing the weights the training proceeded, however my aim is to train with class weights.

I tried to do everything as instructed. One thing I'm not sure about is the number of iterations when training the encoder. I did not see convergence after 45000 iterations(batch size 4). I also tried changing the learning rate until lr = 5e-08, however here is what I have at the "end" of encoder training:

lr = 5e-08
I0720 01:28:11.944630 12740 solver.cpp:228] Iteration 45700, loss = 0.264688
I0720 01:28:11.944663 12740 solver.cpp:244]     Train net output #0: accuracy = 0.755066
I0720 01:28:22.151648 12740 solver.cpp:228] Iteration 45720, loss = 0.20893
I0720 01:28:22.151680 12740 solver.cpp:244]     Train net output #0: accuracy = 0.769897
I0720 01:28:32.375144 12740 solver.cpp:228] Iteration 45740, loss = 0.574258
I0720 01:28:32.375293 12740 solver.cpp:244]     Train net output #0: accuracy = 0.628479
I0720 01:28:42.597137 12740 solver.cpp:228] Iteration 45760, loss = 0.559977
I0720 01:28:42.597182 12740 solver.cpp:244]     Train net output #0: accuracy = 0.703003
I0720 01:28:52.837726 12740 solver.cpp:228] Iteration 45780, loss = 0.27782
I0720 01:28:52.837759 12740 solver.cpp:244]     Train net output #0: accuracy = 0.550903
I0720 01:29:03.087321 12740 solver.cpp:228] Iteration 45800, loss = 0.331952
I0720 01:29:03.087358 12740 solver.cpp:244]     Train net output #0: accuracy = 0.74884

Would you please say, how do you determine the convergence when training the encoder(loss value and difference)? How many iterations did it take? What was the batch size? In case possible, would you please also provide the log file?

When I use the default setting 4 for the "patch", memory is not enough. I have to change it to 1. Then the training of encoder can run and finish.But some per-class-accuracy are zero while other are something like 0.9.
Still keep patch be 1, now after training of encoder-decoder finish, I found all "per-class-accuracy" are zero.
Would you please give me some information for that?
Thanks!

Hi Timo,

Got this when was running this command:
/root/ENet/caffe-enet/build/tools/caffe train -solver /root/ENet/prototxts/enet_solver_encoder.prototxt

I0711 14:14:57.111349 209 net.cpp:100] Creating Layer eltwise2_0_4
I0711 14:14:57.111353 209 net.cpp:434] eltwise2_0_4 <- drop2_0_3
I0711 14:14:57.111371 209 net.cpp:434] eltwise2_0_4 <- bn2_0_4
I0711 14:14:57.111377 209 net.cpp:408] eltwise2_0_4 -> eltwise2_0_4
F0711 14:14:57.111387 209 eltwise_layer.cpp:34] Check failed: bottom[i]->shape() == bottom[0]->shape()
*** Check failure stack trace: ***
@ 0x7f6f05d78daa (unknown)
@ 0x7f6f05d78ce4 (unknown)
@ 0x7f6f05d786e6 (unknown)
@ 0x7f6f05d7b687 (unknown)
@ 0x7f6f0644fbbd caffe::EltwiseLayer<>::Reshape()
@ 0x7f6f06317835 caffe::Net<>::Init()
@ 0x7f6f063186c5 caffe::Net<>::Net()
@ 0x7f6f063296ca caffe::Solver<>::InitTrainNet()
@ 0x7f6f0632a8dc caffe::Solver<>::Init()
@ 0x7f6f0632ac0a caffe::Solver<>::Solver()
@ 0x7f6f062eab23 caffe::Creator_AdamSolver<>()
@ 0x411fd6 caffe::SolverRegistry<>::CreateSolver()
@ 0x40af52 train()
@ 0x40898c main
@ 0x7f6f0456ef45 (unknown)
@ 0x409293 (unknown)
@ (nil) (unknown)
Aborted (core dumped)

What could be wrong?

Hey @TimoSaemann ,
I'm confused as of how to use train Enet on a different dataset. I'm trying to train it on KITTI data set.
Thank You

Hi Timo,

I did as in your tutorial, downloaded the cityscapes_weights.caffemodel, and then ran test_segmentation.py on the test image provided, but the output "prediction" image was all black, and "prediction_rgb" accordingly was of the uniform color. Do you have any ideas what can be wrong?

I ran it like below:
$ python test_segmentation.py --model ENet/prototxts/enet_deploy_final.prototxt \
--weights ENet/enet_weights_zoo/cityscapes_weights.caffemodel
--colours ENet/scripts/cityscapes19.png
--input_image ENet/example_image/munich_000000_000019_leftImg8bit.png
--out_dir ENet/example_image/

I am running the script from the instruction:

python test_segmentation.py --model ../prototxts/enet_deploy_final.prototxt --weights ../enet_weights_zoo/cityscapes_weights.caffemodel --colours ../scripts/cityscapes19.png --input_image ../example_image/munich_000000_000019_leftImg8bit.png --out_dir ../example_image/

by only timing the forward function

t1 = time.time()
out = net.forward_all(**{net.inputs[0]: input_image})
t2 = time.time()
print 'forward time:', (t2-t1) * 1000

forward time: 1605.10015488

am I doing sth. wrong? my GTX 980M should be as good as 30% of a titanic X. So I am expecting at least close to realtime performance?

Thank you

Hi, I tested the speed of your caffemodel on my PC(TITAN X, CUDA8.0, CUDNN5.1), When the resolution of the input image is 512 * 1024, the speed is 1.96s/Frame on cpu and 0.64s/Frame on GPU. There is a big gap with the paper. So, could you update your test result on github? Thanks!

Hi all,

I am doing semantic segmentation for first time.
I had kept cityscapes data to train but the accuracy is 0 throughout & loss is constant 87 with lr = 5e-06.
It has been almost 18 hrs on GTX 980 Ti GPU with 6 GB RAM.
To be frank I am not training whole of cityscapes data but only images from aachen folder.
I have followed all the instructions mentioned in Github page to run this.
I have not changed any of the default settings.

Can anyone guide me to train encoder? I just want to complete the flow of semantic segmentation training as early as possible, So that I can train on my own data set.

I0627 03:28:07.408565 10388 solver.cpp:244] Train net output #20: per_class_accuracy = 0.037037
I0627 03:28:07.408572 10388 sgd_solver.cpp:106] Iteration 74980, lr = 5e-06
I0627 03:28:14.539170 10388 solver.cpp:454] Snapshotting to binary proto file /home/chamcham/Dev/ENet/weights/snapshots_encoder/enet_iter_75000.caffemodel
I0627 03:28:14.571485 10388 sgd_solver.cpp:273] Snapshotting solver state to binary proto file /home/chamcham/Dev/ENet/weights/snapshots_encoder/enet_iter_75000.solverstate
I0627 03:28:14.706820 10388 solver.cpp:317] Iteration 75000, loss = 0.411978
I0627 03:28:14.706852 10388 solver.cpp:322] Optimization Done.
I0627 03:28:14.706856 10388 caffe.cpp:254] Optimization Done.
*** Aborted at 1498501694 (unix time) try "date -d @1498501694" if you are using GNU date ***
PC: @ 0x7fba3dfb813c (unknown)
*** SIGSEGV (@0x0) received by PID 10388 (TID 0x7fba406eda40) from PID 0; stack trace: ***
@ 0x7fba3df6bcb0 (unknown)
@ 0x7fba3dfb813c (unknown)
@ 0x7fba3fef8a87 caffe::Blob<>::~Blob()
@ 0x7fba3ff90ecc caffe::BasePrefetchingDataLayer<>::~BasePrefetchingDataLayer()
@ 0x7fba3ffca7d4 caffe::DenseImageDataLayer<>::~DenseImageDataLayer()
@ 0x7fba3ffca909 caffe::DenseImageDataLayer<>::~DenseImageDataLayer()
@ 0x40e226 std::vector<>::~vector()
@ 0x40ec10 caffe::Net<>::~Net()
@ 0x40edc9 caffe::Net<>::~Net()
@ 0x7fba3ff12690 caffe::AdamSolver<>::~AdamSolver()
@ 0x40d2fe boost::detail::sp_counted_base::release()
@ 0x40b0a7 train()
@ 0x4089ac main
@ 0x7fba3df56f45 (unknown)
@ 0x4092b3 (unknown)
@ 0x0 (unknown)
Segmentation fault (core dumped)

Iam training with NVIDIA Geforce GTX titanZ.
While training, i got stuck with check failed : cudaSuccess(2 Vs 0) out of memory.

Kindly suggest what is the size of GPU needed to train ENet?

Regards
Kumar

Thank you for publishing your code. But why does the algorithm work much better on the cityscape test set images than on random traffic images from the internet? Is there some preprocessing necessary?

In ENet Paper, Its speed is faster than other segmentation algorithm.
But, The processing time of this code 100ms per an image.
Why is it slower than paper?
Which part is bottleneck?
I just wonder this problem.

Thank you. your code gave the much knowledge to me.

I recently downloaded ENet and the Cityscapes dataset in order to train it. Everything worked fine until I enter this command.
ENet/caffe-enet/build/tools/caffe train -solver /ENet/prototxts/enet_solver_encoder.prototxt

I get the following error:
[libprotobuf ERROR google/protobuf/descriptor_database.cc:58] File already exists in database:google/protobuf/descriptor.proto
Here is an image.

I think that this is a problem with either Google Protobuf or OpenCV. I have protobuf version 3.4.0 installed or both python as well as on Ubuntu, but I cannot seem to understand the issue.

On a fresh checkout and downloading directly on dropbox, when running the test_segmentation.py (as shown in the tutorial) (I'm suspecting on the wieghts on dropbox)

W0913 00:07:53.826993 10758 _caffe.cpp:142] Net('../prototxts/enet_deploy_final.prototxt', 1, weights='../enet_weights_zoo/cityscapes_weights.caffemodel')
[libprotobuf ERROR google/protobuf/text_format.cc:291] Error parsing text-format caffe.NetParameter: 65:12: Message type "caffe.LayerParameter" has no field named "bn_param".
F0913 00:07:53.831104 10758 upgrade_proto.cpp:88] Check failed: ReadProtoFromTextFile(param_file, param) Failed to parse NetParameter file: ../prototxts/enet_deploy_final.prototxt
*** Check failure stack trace: ***
Aborted (core dumped)

Could you check the dropbox files?

Thanks for the awesome work!

thx!

Hi, I just finished compiling ENet and trying to use the pre-trained weights by running "sh cityscapes_weights.sh" however I think dropbox is not responding:

and I got this:

--2017-10-02 15:00:20-- https://www.dropbox.com/sh/5wjmb5twfr5b0wo/AADEpW5a8-GSSt5pfJYCKxoOa?dl=1
Resolving www.dropbox.com... 46.82.174.68, 162.125.82.1, 2620:100:6032:1::a27d:5201
Connecting to www.dropbox.com|46.82.174.68|:443... failed: Operation timed out.
Connecting to www.dropbox.com|162.125.82.1|:443... failed: Operation timed out.
Connecting to www.dropbox.com|2620:100:6032:1::a27d:5201|:443... failed: No route to host.
--2017-10-02 15:02:54-- https://www.dropbox.com/s/3i367gsl7sspeo1/cityscapes_weights_before_bn_merge.caffemodel?dl=1
Resolving www.dropbox.com... 93.46.8.89
Connecting to www.dropbox.com|93.46.8.89|:443... failed: Operation timed out.
Retrying.

so is it because of my network configuration or something else ? I need help, please.

Hi, do you plan to release code with cudnn v6, it is supposed to be faster than cudnnv5.1. Thanks.

While running the command
./caffe-enet/build/tools/caffe train -solver ./prototxts/enet_solver_encoder.prototxt

I am getting the following error.

Traceback (most recent call last):
File "/home/madan/git_repos/ENet/caffe-enet/python/spatial_dropout.py", line 3, in
import caffe, json
File "/home/madan/git_repos/ENet/caffe-enet/python/caffe/init.py", line 1, in
from .pycaffe import Net, SGDSolver, NesterovSolver, AdaGradSolver, RMSPropSolver, AdaDeltaSolver, AdamSolver
File "/home/madan/git_repos/ENet/caffe-enet/python/caffe/pycaffe.py", line 15, in
import caffe.io
File "/home/madan/git_repos/ENet/caffe-enet/python/caffe/io.py", line 8, in
from caffe.proto import caffe_pb2
File "/home/madan/git_repos/ENet/caffe-enet/python/caffe/proto/caffe_pb2.py", line 853, in
options=_descriptor._ParseOptions(descriptor_pb2.FieldOptions(), _b('\020\001')), file=DESCRIPTOR),
TypeError: init() got an unexpected keyword argument 'file'

I have compiled protobuf from source.
Here's my result from command 'which protoc'
/usr/local/bin/protoc

My current version is 3.5.0.
Please help guys. I have already spent half day working on this.

[EDIT]
I rebuild the protobuf with version 3.3 and it worked.
Thanks

I have one issue while training

I0728 12:07:31.305574 32224 layer_factory.hpp:77] Creating layer drop1_0_3
*** Aborted at 1501232851 (unix time) try "date -d @1501232851" if you are using GNU date ***
PC: @ 0x7f9a8456db73 std::_Hashtable<>::clear()
*** SIGSEGV (@0x9) received by PID 32224 (TID 0x7f9aead65ac0) from PID 9; stack trace: ***
@ 0x7f9ae83b84b0 (unknown)
@ 0x7f9a8456db73 std::_Hashtable<>::clear()
@ 0x7f9a8455eca6 google::protobuf::DescriptorPool::FindFileByName()
@ 0x7f9a8453bdc8 google::protobuf::python::cdescriptor_pool::AddSerializedFile()
@ 0x7f9ae8ff07d0 PyEval_EvalFrameEx
@ 0x7f9ae911901c PyEval_EvalCodeEx
@ 0x7f9ae906f3dd (unknown)
@ 0x7f9ae90421e3 PyObject_Call
@ 0x7f9ae9062ae5 (unknown)
@ 0x7f9ae8ff9123 (unknown)
@ 0x7f9ae90421e3 PyObject_Call
@ 0x7f9ae8fed13c PyEval_EvalFrameEx
@ 0x7f9ae911901c PyEval_EvalCodeEx
@ 0x7f9ae8fe7b89 PyEval_EvalCode
@ 0x7f9ae907c1b4 PyImport_ExecCodeModuleEx
@ 0x7f9ae907cb8f (unknown)
@ 0x7f9ae907e300 (unknown)
@ 0x7f9ae907e5c8 (unknown)
@ 0x7f9ae907f6db PyImport_ImportModuleLevel
@ 0x7f9ae8ff6698 (unknown)
@ 0x7f9ae90421e3 PyObject_Call
@ 0x7f9ae9118447 PyEval_CallObjectWithKeywords
@ 0x7f9ae8feb5c6 PyEval_EvalFrameEx
@ 0x7f9ae911901c PyEval_EvalCodeEx
@ 0x7f9ae8fe7b89 PyEval_EvalCode
@ 0x7f9ae907c1b4 PyImport_ExecCodeModuleEx
@ 0x7f9ae907cb8f (unknown)
@ 0x7f9ae907e300 (unknown)
@ 0x7f9ae907e5c8 (unknown)
@ 0x7f9ae907f6db PyImport_ImportModuleLevel
@ 0x7f9ae8ff6698 (unknown)
@ 0x7f9ae90421e3 PyObject_Call

and I can correctly import caffe
any help? Thanks.

Hi, I tried to run an example on the python and everything turned out fine. But when I run an example on c ++, the result does not match the declared one.

The console displays:
Processing time = 361 ms
Output_blob (n, c, h, w) = 1, 19, 512, 1024
Processing time on the python is 1800 ms.

If I use weights with Segnet, then I get a fine result, but the speed has not changed.

Hi, I have try to training using multi-gpu . I got this error

[python_layer.hpp:25] PythonLayer is not implemented in Multi-GPU training

So, does any one have experience on change the code spatial_dropout.py to use multi-gpu ?

Hai,

During training of an encoder iam met with Import error:no module named spatial_dropout.
Kindly suggest me how to solve this issue since iam under job pressure to get some decent results.
Kindly do the needful.

Regards
Kumar

Thx for your great work.
I am doing fine with your model by now for my binary segmantation task.
But at the edges, when I applied the softmax, the smaller part become slightly larger than expected but when there is no softmax, holes appeared in the smaller part. That's what I can understand.

But about the dilatation part, I saw you are utilizing 2 4 8 16 for the 4 layers. Is it reasonable to decrease them for better edge performance. I know the dilated convolution is for larger receptive field without losing pixels. But I guess the detail is lost on that procedure. So is it reasonable like I am setting all the dilation to 2 or 0? What's pros and cons? Thx

Lastly, it's weird my gpu utilzation is low like 30 to 40% and power assumption only 90W on my 1080ti.
While when I was training with tensorflow, 99% 270W and the card is like a furnace.
Any idea on the low usage of the card?

By the way, it seems there are no normalization with the raw pics like mean and variance normalization, am I right?

Thx

I0626 19:33:00.715394 10046 net.cpp:100] Creating Layer bn1_0_2
I0626 19:33:00.715397 10046 net.cpp:434] bn1_0_2 <- conv1_0_2
I0626 19:33:00.715414 10046 net.cpp:408] bn1_0_2 -> bn1_0_2
I0626 19:33:00.715584 10046 net.cpp:150] Setting up bn1_0_2
I0626 19:33:00.715590 10046 net.cpp:157] Top shape: 4 64 128 256 (8388608)
I0626 19:33:00.715593 10046 net.cpp:165] Memory required for data: 394657792
I0626 19:33:00.715602 10046 layer_factory.hpp:77] Creating layer drop1_0_3
ImportError: No module named spatial_dropout
*** Aborted at 1498473180 (unix time) try "date -d @1498473180" if you are using GNU date ***
PC: @ 0x7f5b31ba7a95 caffe::Blob<>::~Blob()
*** SIGSEGV (@0x7f5b321d02b8) received by PID 10046 (TID 0x7f5b3239ca40) from PID 840762040; stack trace: ***
@ 0x7f5b2fc1acb0 (unknown)
@ 0x7f5b31ba7a95 caffe::Blob<>::~Blob()
@ 0x7f5b31c3fed8 caffe::BasePrefetchingDataLayer<>::~BasePrefetchingDataLayer()
@ 0x7f5b31c797d4 caffe::DenseImageDataLayer<>::~DenseImageDataLayer()
@ 0x7f5b31c79909 caffe::DenseImageDataLayer<>::~DenseImageDataLayer()
@ 0x40e226 std::vector<>::~vector()
@ 0x7f5b31c1f098 caffe::Net<>::Net()
@ 0x7f5b31bb7cea caffe::Solver<>::InitTrainNet()
@ 0x7f5b31bb8efc caffe::Solver<>::Init()
@ 0x7f5b31bb922a caffe::Solver<>::Solver()
@ 0x7f5b31bc2b33 caffe::Creator_AdamSolver<>()
@ 0x411ff6 caffe::SolverRegistry<>::CreateSolver()
@ 0x40af72 train()
@ 0x4089ac main
@ 0x7f5b2fc05f45 (unknown)
@ 0x4092b3 (unknown)
@ 0x0 (unknown)
Segmentation fault (core dumped)

/usr/bin/ld: cannot find -lopencv_dep_cudart

Did anyone met with same error??

How to train using multiple gpu?

I am trying to train ENet with a multichannel dataset, input image size is 320x384. I get the following error on training,
I1007 10:52:27.313614 10208 layer_factory.hpp:77] Creating layer conv5_0_4
I1007 10:52:27.313629 10208 net.cpp:100] Creating Layer conv5_0_4
I1007 10:52:27.313638 10208 net.cpp:434] conv5_0_4 <- prelu4_2_4_prelu4_2_4_0_split_1
I1007 10:52:27.313649 10208 net.cpp:408] conv5_0_4 -> conv5_0_4
I1007 10:52:27.313881 10208 net.cpp:150] Setting up conv5_0_4
I1007 10:52:27.313894 10208 net.cpp:157] Top shape: 3 16 96 80 (368640)
I1007 10:52:27.313900 10208 net.cpp:165] Memory required for data: 971200512
I1007 10:52:27.313910 10208 layer_factory.hpp:77] Creating layer bn5_0_4
I1007 10:52:27.313928 10208 net.cpp:100] Creating Layer bn5_0_4
I1007 10:52:27.313937 10208 net.cpp:434] bn5_0_4 <- conv5_0_4
I1007 10:52:27.313947 10208 net.cpp:408] bn5_0_4 -> bn5_0_4
I1007 10:52:27.314162 10208 net.cpp:150] Setting up bn5_0_4
I1007 10:52:27.314174 10208 net.cpp:157] Top shape: 3 16 96 80 (368640)
I1007 10:52:27.314180 10208 net.cpp:165] Memory required for data: 972675072
I1007 10:52:27.314190 10208 layer_factory.hpp:77] Creating layer upsample5_0_4
I1007 10:52:27.314200 10208 net.cpp:100] Creating Layer upsample5_0_4
I1007 10:52:27.314208 10208 net.cpp:434] upsample5_0_4 <- bn5_0_4
I1007 10:52:27.314218 10208 net.cpp:434] upsample5_0_4 <- pool1_0_4_mask
I1007 10:52:27.314229 10208 net.cpp:408] upsample5_0_4 -> upsample5_0_4
I1007 10:52:27.314239 10208 upsample_layer.cpp:27] Params 'pad_out_{}_' are deprecated. Please declare upsample height and width useing the upsample_h, upsample_w parameters.
F1007 10:52:27.314256 10208 upsample_layer.cpp:58] Check failed: bottom[0]->channels() == bottom[1]->channels() (16 vs. 21)

I am trying to use the same architecture as your default prototxt, enet_train_encoder_decoder.prototxt. I am not sure why the upsample5_0_4 layer is expecting a 21 channel input.