ljanyst / image-segmentation-fcn Goto Github PK

View Code? Open in Web Editor NEW

87.0 9.0 33.0 3.21 MB

Semantic Image Segmentation using a Fully Convolutional Neural Network in TensorFlow

Home Page: http://jany.st/post/2017-06-25-semantic-image-segmentation-using-fcns.html

Python 100.00%

tensorflow image-segmentation fully-convolutional-networks vgg16

image-segmentation-fcn's Introduction

Semantic Image Segmentation using a Fully Convolutional Neural Network

Overview

The programs in this repository train and use a fully convolutional neural network to take an image and classify its pixels. The network is transfer-trained basing on the VGG-16 model using the approach described in this paper by Jonathan Long et al. The software is generic and easily extendable to any dataset, although I only tried with KITTI Road Dataset and Cityscapes dataset so far. All you need to do to introduce a new dataset is to create a new source_xxxxxx.py file defining your dataset. The definition is a class that contains seven attributes:

image_size - self-evident, both horizontal and vertical dimention need to be divisible by 32
num_classes - number of classes that the model is supposed to handle
label_colors - a dictionary mapping a class number to a color; used for blending of the classification results with input image
num_training - number of training samples
num_validation - number of validation samples
train_generator - a generator producing training batches
valid_generator - a generator producing validation batches

See source_kitti.py or source_cityscapes.py for a concrete example. The trainer picks the source based on the value of the --data-source parameter.

The KITTI dataset

Training the model on the KITTI Road Dataset essentially means that infer.py will be able to take images from a car's dashcam and paint the road pink. It generalizes fairly well even to pretty complicated cases:

The model that produced the above images was trained for 500 epochs on the images contained in this zip file. The training program fills tensorboard with the loss summary and a sneak peek of the current performance on validation examples. The top row contains the ground truth and the bottom one the network's output.

The Cityscapes dataset

This dataset is more complex than the previous one. It has fine image annotations for 29 classes of objects. The images are video frames taken in German cities and there is around 11GB of them.

The model that produced the images was trained for 150 epochs.

image-segmentation-fcn's People

Contributors

Stargazers

Watchers

image-segmentation-fcn's Issues

Steps to run the model?

How exactly do we need to run the files?
can you please give the step wise explanation?

height and width dimensions

Why the height and the width of the images have to be divisible by 32? Is it due to the upsampling operations?

Can we test this code on images different from that of training and validation images?

Different Sized Input Image Error

I used the default set image size (576x160) with batch size = 10 with my own dataset to compensate for my hardware and it trained fine.
But when i set my own image size, for e.g.(206x 293), i get the following error:

InvalidArgumentError (see above for traceback): Incompatible shapes: [10,304,208,2] vs. [10,320,224,2]

Traceback (most recent call last):
  File "C:\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1323, in _do_call
    return fn(*args)
  File "C:\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1302, in _run_fn
    status, run_metadata)
  File "C:\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\framework\errors_impl.py", line 473, in __exit__
    c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: Incompatible shapes: [10,304,208,2] vs. [10,320,224,2]
         [[Node: sum/Add = Add[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](sum/mul, sum/mul_1)]]
         [[Node: reshape/Mean/_311 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_1596_reshape/Mean", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "train.py", line 132, in <module>
    loss_batch, _ = sess.run([loss, optimizer], feed_dict=feed)
  File "C:\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 889, in run
    run_metadata_ptr)
  File "C:\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1120, in _run
    feed_dict_tensor, options, run_metadata)
  File "C:\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1317, in _do_run
    options, run_metadata)
  File "C:\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1336, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Incompatible shapes: [10,304,208,2] vs. [10,320,224,2]
         [[Node: sum/Add = Add[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](sum/mul, sum/mul_1)]]
         [[Node: reshape/Mean/_311 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_1596_reshape/Mean", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

Caused by op 'sum/Add', defined at:
  File "train.py", line 84, in <module>
    net.build_from_vgg(args.vgg_dir, source.num_classes, progress_hook='tqdm')
  File "H:\Documents (H-drive)\FAST\FYP\image-segmentation-fcn-master\fcnvgg.py", line 68, in build_from_vgg
    self.__make_result_tensors()
  File "H:\Documents (H-drive)\FAST\FYP\image-segmentation-fcn-master\fcnvgg.py", line 143, in __make_result_tensors
    tf.add(2*vgg4_reshaped, 4*vgg7_reshaped))
  File "C:\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\ops\gen_math_ops.py", line 182, in add
    "Add", x=x, y=y, name=name)
  File "C:\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "C:\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\framework\ops.py", line 2956, in create_op
    op_def=op_def)
  File "C:\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\framework\ops.py", line 1470, in __init__
    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access

InvalidArgumentError (see above for traceback): Incompatible shapes: [10,304,208,2] vs. [10,320,224,2]
         [[Node: sum/Add = Add[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](sum/mul, sum/mul_1)]]
         [[Node: reshape/Mean/_311 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_1596_reshape/Mean", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

Is there something in the code that i am forgetting to change?

Problem in *label_list

Very great work!
When I run the codes with cityscapes dataset, a problem occured, at line 136 in source_cityscapes.py
label_all = np.dstack([label_bg, *label_list])
^SyntaxError: invalid syntax

How can i fix this problem? i appreciate your reply.

Retrain model from checkpoint

Hello,
i want retrain my model from a checkpoint.
So, starting from train.py file, instead of calling the build_from_vgg function, i called build_from_metagraph function (passing to the function last checkpoint) , then i called net.get_optimizer(labels) and in the function i changed the variable self.numclasses with a number with number of classes (in my case is 2).

When i run my code I get this exception:
ValueError: Duplicate node name in graph: 'optimizer/conv1_1/filter/Adam'
when the line optimizer = optimizer.minimize(loss) is called in fcnvgg.py file.

Could you help me pls?

regarding enableing your code to handle different image size

Hello Lukasz,

It is very nice to see your code. I have a question, if the training images are of different size, what kind of modifications that I have to make on your code? Thanks a lot.

regarding the number of classes and data augmentation mechanism

Hi Lukasz, thank you very much for sharing the solution. I have two questions:

In some ground truth images, I found some pixels are marked as black. What do those pixels represent. My understanding is that this should be a binary semantic segmentation problem, i.e., we only have two classes, background and road. Moreover, how do you handle those black pixels?

What kind of data augmentation mechanisms that you have been using?

Thanks a lot for your response.