Tensorflow implementation of "Semantic Instance Segmentation with a Discriminative Loss Function"

License: MIT License

Python 100.00%

tusimple-dataset instance-segmentation image-recognition tensorflow semantic-segmentation clustering

instance-segmentation-with-discriminative-loss-tensorflow's Introduction

Instance Segmentation with a Discriminative Loss Function

Tensorflow implementation of Semantic Instance Segmentation with a Discriminative Loss Function trained on the TuSimple dataset

Files

├── data here the data should be stored
│ └── tusimple_dataset_processing.py processes the TuSimple dataset
├── doc documentation
├── inference_test inference related data
│ └── images for testing the inference
├── trained_model pretrained model for finetuning
├── clustering.py mean-shift clustering
├── datagenerator.py feeds data for training and evaluation
├── enet.py Enet architecture
├── inference.py tests inference on images
├── loss.py defines discriminative loss function
├── README.md
├── training.py contains training pipeline
├── utils.py contains utilities files for building and initializing the graph
└── visualization.py contains visualization of the clustering and pixel embeddings

Instructions

Inference

To test the inference of the trained model execute:
python inference.py --modeldir trained_model --outdir inference_test/results

Training

Download the TuSimple training dataset and extract its contents to the data folder. The folder structure should look like this:
| data
├── train_set
│     ├── clips
│     ├── label_data_0313.json
│     ├── label_data_0531.json
│     ├── label_data_0601.json
│     └── readme.md
Run the following script to prepare images and labels.
python data/tusimple_dataset_processing.py <train_data_dir>
This should create the following images and labels folders:
| data
├── train_set
├── images
└── labels
For training on the dataset execute:
python training.py
alternatively use optional parameters (default parameters in this example):
python training --srcdir data --modeldir pretrained_semantic_model --outdir saved_model --logdir log --epochs 50 --var 1.0 --dist 1.0 --reg 1.0 --dvar 0.5 --ddist 1.5
To test the trained network execute: python inference.py --modeldir saved_model

Training Pipeline

Training Visualization

Feature space projection of one image for consecutive gradient steps. Each point represents one pixel embedding and each color represents an instance in the label.

Results

Todo

pip requirements
semantic segmentation code
visualization

Tensorflow version 1.2

Reference and Credits

This application uses Open Source components. We acknowledge and are grateful to these developers for their contributions to open source:

Project: TensorFlow-ENet https://github.com/kwotsin/TensorFlow-ENet
Project: TuSimple dataset http://benchmark.tusimple.ai
Project: Fast Scene Understanding https://github.com/DavyNeven/fastSceneUnderstanding

Related work

Paper: Towards End-to-End Lane Detection: an Instance Segmentation Approach

instance-segmentation-with-discriminative-loss-tensorflow's People

Contributors

Stargazers

Watchers

instance-segmentation-with-discriminative-loss-tensorflow's Issues

tf.unsorted_segment_sum too slow?

In my experiments (a different setup but using discriminative loss), I am finding the loss computation to be very time-consuming. I suspect the lag is coming from the tf.unsorted_segment_sum. Have you noticed such latency issues in your experiments? Is there perhaps a way to speed up/optimize the loss computation?

Calculating Accuracy, FP and FN

hq-jiang
How do we calculate the Tusimple evaluation metrics on the test data with your model. Please assist

Requirements

Can you please list the requirements for the installation?

How do I change the batch_size?

Hi, I am Dongwhan Lee from Korea.

I am impressed by your work and trying to follow yours.
However, I encountered the error like below:

That error message appears after I changed batch_size=1 to batch_size=16 in "training.py".
I looked into "datagenerator.py" too but failed to how to fix the error.

Is there any way to change the batch_size? If so, could you let me know?
Plus, I also wonder where that 1 in (1, 512, 512, 3) comes from.

Thank you in advance.

semantic_segmentation code

Hello ,thanks a lot for your code . In my experiment，the inference time is about 60ms ,but the cluster time is more than 4s. so I only want to do the binary segmentation. Is there complete code for semantic segmentation? I got a trained model by your 'transfer_semantic.py' and failed in writing the inference and evaluation code.
Have a nice day!

paper issue

hello ,i have read the paper"Towards End-to-End Lane Detection: an Instance Segmentation Approach"
do you know what's the meaning of the "bounded inverse class weighting"(to solve the problem that the lane lines area is too small). i can not find the detailed explanation about the "bounded inverse class weighting"
thanks so much!

No module named symeig

upon
/instance-segmentation-with-discriminative-loss-tensorflow ]$ python2 inference.py --modeldir trained_model --outdir inference_test/results
I get the error:

Traceback (most recent call last):

  File "inference.py", line 12, in <module>
    from clustering import cluster, get_instance_masks, save_instance_masks
   from sklearn.cluster import MeanShift, estimate_bandwidth

  File "/usr/lib/python2.7/dist-packages/sklearn/cluster/__init__.py", line 6, in <module>
    from .spectral import spectral_clustering, SpectralClustering

  File "/usr/lib/python2.7/dist-packages/sklearn/cluster/spectral.py", line 17, in <module>
    from ..manifold import spectral_embedding

  File "/usr/lib/python2.7/dist-packages/sklearn/manifold/__init__.py", line 8, in <module>
    from .spectral_embedding_ import SpectralEmbedding, spectral_embedding

  File "/usr/lib/python2.7/dist-packages/sklearn/manifold/spectral_embedding_.py", line 12, in <module>
    from scipy.sparse.linalg.eigen.lobpcg.lobpcg import symeig

ImportError: cannot import name symeig

symeig is not a module in scipy.sparse.linalg.eigen.lobpcg.lobpcg.

I guess there must be some edits required

training question

@hq-jiang Thank you for your code, I am confused about your training pipeline. Do you means training step by step? And how to do it ?

error info

Hi, I run: python inference.py --modeldir trained_model --outdir inference_test/results
What is the problem?
tensorflow:1.4
thanks!
error info:
InvalidArgumentError (see above for traceback): Assign requires shapes of both tensors to match. lhs shape= [2,2,1,16] rhs shape= [2,2,3,16]
[[Node: save/Assign_538 = Assign[T=DT_FLOAT, _class=["loc:@Instance/transfer_layer/conv2d_transpose/weights"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](Instance/transfer_layer/conv2d_transpose/weights, save/RestoreV2_538/_3)]]

pretrained model for semantic segmentation

Hi @hq-jiang,

Nice work for discriminative loss.
I am trying to use your code for other data such as cityscapes or CVPPP (leaf segmentation) as discussed in the original paper. I would like to ask whether you can release code model for semantic segmentation as well? I would like to train from scratch in those data and therefore a separate training for semantic mask is required.

Bests,

InvalidArgumentError

InvalidArgumentError (see above for traceback): indices[262144] = [1, 128, 0, 0] does not index into shape [2,128,128,64]
[[node ENet/unpool/ScatterNd (defined at /home/lunana/instance-segmentation-with-discriminative-loss-tensorflow/enet.py:110) ]]

how to set the num_classes?

hello, than you for your wonderful work! I try your work to my dataset, but there is a question. my picture is 5~18 lane line per image ,so I draw my lane line in 19 different colors. and you use 6 different colors，the num_classer is 12. but when I change my num_classes like you in 38. There are some mistakes. but when i change it to 12.Training can be done successfully . So how do I set up this parameter？
Can you give me some advice？
thx

error in utils.load_enet

Hello, i really appreciate your work. It helped me a lot.
but i met some problems when i ran "python training.py",it said as follow,

Traceback (most recent call last):
File "training.py", line 241, in
run()
File "training.py", line 115, in run
last_prelu = utils.load_enet(sess, model_dir, input_image, batch_size)
File "/home/lps/DL/instance-segmentation-with-discriminative-loss-tensorflow/utils.py", line 27, in load_enet
saver.restore(sess, checkpoint)
File "/home/lps/anaconda2/envs/tensorflow/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1439, in restore
{self.saver_def.filename_tensor_name: save_path})
File "/home/lps/anaconda2/envs/tensorflow/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 767, in run
run_metadata_ptr)
File "/home/lps/anaconda2/envs/tensorflow/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 965, in _run
feed_dict_string, options, run_metadata)
File "/home/lps/anaconda2/envs/tensorflow/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1015, in _do_run
target_list, options, run_metadata)
File "/home/lps/anaconda2/envs/tensorflow/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1035, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.NotFoundError: Key ENet/bottleneck3_7_batch_norm2/gamma not found in checkpoint
[[Node: save/RestoreV2_473 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_recv_save/Const_0, save/RestoreV2_473/tensor_names, save/RestoreV2_473/shape_and_slices)]]
[[Node: save/RestoreV2_619/_1241 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_3105_save/RestoreV2_619", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"]]

Caused by op u'save/RestoreV2_473', defined at:
File "training.py", line 241, in
run()
File "training.py", line 115, in run
last_prelu = utils.load_enet(sess, model_dir, input_image, batch_size)
File "/home/lps/DL/instance-segmentation-with-discriminative-loss-tensorflow/utils.py", line 26, in load_enet
saver = tf.train.Saver(variables_to_restore)
File "/home/lps/anaconda2/envs/tensorflow/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1051, in init
self.build()
File "/home/lps/anaconda2/envs/tensorflow/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1081, in build
restore_sequentially=self._restore_sequentially)
File "/home/lps/anaconda2/envs/tensorflow/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 675, in build
restore_sequentially, reshape)
File "/home/lps/anaconda2/envs/tensorflow/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 402, in _AddRestoreOps
tensors = self.restore_op(filename_tensor, saveable, preferred_shard)
File "/home/lps/anaconda2/envs/tensorflow/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 242, in restore_op
[spec.tensor.dtype])[0])
File "/home/lps/anaconda2/envs/tensorflow/lib/python2.7/site-packages/tensorflow/python/ops/gen_io_ops.py", line 668, in restore_v2
dtypes=dtypes, name=name)
File "/home/lps/anaconda2/envs/tensorflow/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 763, in apply_op
op_def=op_def)
File "/home/lps/anaconda2/envs/tensorflow/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2395, in create_op
original_op=self._default_original_op, op_def=op_def)
File "/home/lps/anaconda2/envs/tensorflow/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1264, in init
self._traceback = _extract_stack()

NotFoundError (see above for traceback): Key ENet/bottleneck3_7_batch_norm2/gamma not found in checkpoint
[[Node: save/RestoreV2_473 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_recv_save/Const_0, save/RestoreV2_473/tensor_names, save/RestoreV2_473/shape_and_slices)]]
[[Node: save/RestoreV2_619/_1241 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_3105_save/RestoreV2_619", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"]]

it seems some errors in Enet checkpoint files. I hope you can help me. Thanks!

Clustering in discriminative loss

In the original paper https://arxiv.org/pdf/1708.02551.pdf they are used only pixels of concrete class (without background) and then do instance segmentation.
But in discriminative_loss.py you calculate loss for full embedding tensor including background:
unique_labels, unique_id, counts = tf.unique_with_counts(correct_label) counts = tf.cast(counts, tf.float32) num_instances = tf.size(unique_labels)
Here num_instances here = num of lines + 1 (background class)
Am I right?

nan loss

Have you ever experienced a nan loss when training?

l_dist term when instance_num=1

I am using you discriminative loss function for another use. And I discovered that if instance_num==1, the l_dist term will become nan because mu_norm=[]. So I suggest to add a sentence like l_dist = tf.cond(num_instances<=1, lambda:tf.zeros(1, dtype=tf.float32), lambda:tf.reduce_mean(mu_norm)) after the calculation of l_dist term.

Do you think it's necessary? If so, may I add a pull request to your repo?

How to apply multi-class instance segmentation with the discriminative loss function?

Hello, I'm working on a project that needs to detect multiple classes of traffic markings(normal traffic lane marking, arrows, zebra crossing,etc.), so I need to apply a discriminative loss function for multi-class situation where there could still be multiple instancess in each class. I checked your code and as far as I can tell ,it only applies to binary segementation. Any advice about how I can upgrade the loss function such that it 'll work for multi-class case? Thank you very much

where download pretrained model?

where download weights files? thanks!

hq-jiang / instance-segmentation-with-discriminative-loss-tensorflow Goto Github PK