pierluigiferrari / ssd_keras Goto Github PK

Hi,
First off, this is a great project with great documentation! I just had a few questions though. First some preliminaries:

1. Successfully able to train using ssd7 notebook on your udacity data set
2. Running on Ubuntu 16.10 with GTX 1060

Questions/Notes:

1. I am going to be training on my own data set. I have already put the format of the labels as you desire, and it seems to be working. For example the plot that shows the ground truth at the end looks as expected (of course the predictions stink, thats why I'm here!).

2. My input resolution size is 400 X 727 (actual images are 1914 X 1042). Reading the documentation I made sure to set the resize parameter to (400,727) for all the generators. Is there any issue that should occur with this resolution? Anywhere else I should be mindful to tweak because of the unusual resolution.

3. I noticed that during training (10 epochs, 6,000 images per epoch), that the loss was going down as well as the the validation loss.

4. The actual shape of the output of my network is float32 (21796, 35), yours was (10316, 18). Think its just from the image resolution size.

5. After training (10 epochs), I tried out the predictions. They were all over the place, and there were more than 100. In fact I noticed that the classification (supposed to be an integer right?) looked like it was a floating point value, like 1.2 instead of 1.

6. After saving the weights, If I restart jupyter and reload the weights (load_weights), I cant reproduce the predictions generated from decode_y2. The actual predictions from the model are still there (10316,18), but decode_y2 returns an empty array.

7. I am trying to also the load the model, but a running into this error:

model = load_model('./models/ssd7_gtav_0.h5', custom_objects={"AnchorBoxes": AnchorBoxes, "L2Normalization": L2Normalization})
4 variance values must be pased, but 2 values were received.

Anyways, any help would be appreciated!

For example, in this repo, image translation is adopted, which is common in image classification, but is it necessary for object detection?
Why not realize the original Data aumentation in SSD?

I changed the batch size (x2) in the middle of training and the loss cut in half with no performance improvement. Is this accepted behavior? I then have to do bookkeeping to keep track of this hyper-parameter to determine if it performs well.

Sorry ,I'm s a rookie in keras & Neural network.
Pardom,can I ask you a question?

I complete the model training and save model & weights，but when I want to load model next time ，
there will be mistakes : ValueError: Unknown layer: AnchorBoxes
#load model && load weights model = load_model('./model/ssd7_0.h5') model.load_weights('./model/ssd7_0_weights.h5')
I really want to know how to solve this problem，would like to ask about the experienced elders.
thankes!

I have written a mAP code using your code. Should I add PR?
Also, I am training using our own classifier model as base model. However, my model loss goes down only till 10-12 Epochs for pascal VOC. Do you think that 10-12 epochs is sufficient for Pascal voc, especially since my base model is very light weight not as heavy as VGG16.

Hi,
I am new to image classification. I tried out the train_ssd300 code exactly the way you implemented it (same VOC datasets, parameters), the only difference is that I had to use decode_y instead of decode_y2 because the latter output an empty matrix each time. When I run the code for 10epochs I get the image with both ground truth and prediction boxes bounding the whole images. Am I missing something? In addition the prediction is completely off and has a low percent. I thought that it was because I didn't train the model enough, but it won't train longer than 7 epochs. Any help would be great!

when I train the ssd7 model, I download the consolidated and resized dataset you used , I can't find val_labels.csv.

hello,
i want to load model with model = load_model('./ssd7_0.h5') (ie. uncomment the line)
i got error after some search, i add
model = load_model('./ssd7_0.h5', custom_objects={'AnchorBoxes': AnchorBoxes})
but i got another error
TypeError: __init__() missing 4 required positional arguments: 'img_height', 'img_width', 'this_scale', and 'next_scale'

i try another way : trying to create model by code and load weights with model.load_weights('./ssd7_0_weights.h5') ... but finish with ValueError: Dimension 0 in both shapes must be equal, but are 3 and 12 for 'Assign_42' (op: 'Assign') with input shapes: [3,3,64,24], [12,64,3,3].

is load model or weights operational ? or need add some code (get_config in AnchorBoxes, ...)

best regard

(First of course, this is an amazing piece of code, thank you)

In the compute_loss function:

# 1: Compute the losses for class and box predictions for every box

        classification_loss = tf.to_float(self.log_loss(y_true[:,:,:-12], y_pred[:,:,:-12])) # Output shape: (batch_size, n_boxes)
        localization_loss = tf.to_float(self.smooth_L1_loss(y_true[:,:,-12:-8], y_pred[:,:,-12:-8])) # Output shape: (batch_size, n_boxes)

I'm probably wrong, but if there are say 6 classes, then the last dimension of y_true should be 6 one-hot encoded classes + 4 ground truth box coordinates + 4 arbitrary entries = 14. And so by indexing y_true[:,:,-12] we enter the log_loss function only the first 2 one-hot encoded classes, no?

If there were only 2 classes this would throw an error.

Shouldn't it be in fact y_true[:, :, -8]?

Same with localization_loss which should take the 4 box coordinates, shouldn't it be y_true[:,:,-8:-4]?

(Sorry if I'm completely wrong)

Just like drorhilman my model is not learning in a two class situation. The labels seem te be correct (I wrote a new parser outputs for example [[1, 571, 327, 479, 181]] and the correct file name). The loss starts at inf but after one batch it drops to 0 and stays there. The model however does not learn. Any ideas?

While using my own dataset which is in VOC format, I end up with this error while initiating the training.

Epoch 1/10

ValueErrorTraceback (most recent call last)
in ()
21 cooldown=0)],
22 validation_data = val_generator,
---> 23 validation_steps = ceil(n_val_samples/batch_size))
24
25

/nfs/p4/ceusers/vinod/.venv2.7-gpu/local/lib/python2.7/site-packages/keras/legacy/interfaces.pyc in wrapper(*args, **kwargs)
85 warnings.warn('Update your ' + object_name + 86 ' call to the Keras 2 API: ' + signature, stacklevel=2)
---> 87 return func(*args, **kwargs)
88 wrapper._original_function = func
89 return wrapper

/nfs/p4/ceusers/vinod/.venv2.7-gpu/local/lib/python2.7/site-packages/keras/engine/training.pyc in fit_generator(self, generator, steps_per_epoch, epochs, verbose, callbacks, validation_data, validation_steps, class_weight, max_queue_size, workers, use_multiprocessing, shuffle, initial_epoch)
2075 outs = self.train_on_batch(x, y,
2076 sample_weight=sample_weight,
-> 2077 class_weight=class_weight)
2078
2079 if not isinstance(outs, list):

/nfs/p4/ceusers/vinod/.venv2.7-gpu/local/lib/python2.7/site-packages/keras/engine/training.pyc in train_on_batch(self, x, y, sample_weight, class_weight)
1789 sample_weight=sample_weight,
1790 class_weight=class_weight,
-> 1791 check_batch_axis=True)
1792 if self.uses_learning_phase and not isinstance(K.learning_phase(), int):
1793 ins = x + y + sample_weights + [1.]

/nfs/p4/ceusers/vinod/.venv2.7-gpu/local/lib/python2.7/site-packages/keras/engine/training.pyc in _standardize_user_data(self, x, y, sample_weight, class_weight, check_batch_axis, batch_size)
1407 self._feed_input_shapes,
1408 check_batch_axis=False,
-> 1409 exception_prefix='input')
1410 y = _standardize_input_data(y, self._feed_output_names,
1411 output_shapes,

/nfs/p4/ceusers/vinod/.venv2.7-gpu/local/lib/python2.7/site-packages/keras/engine/training.pyc in _standardize_input_data(data, names, shapes, check_batch_axis, exception_prefix)
140 ' to have ' + str(len(shapes[i])) +
141 ' dimensions, but got array with shape ' +
--> 142 str(array.shape))
143 for j, (dim, ref_dim) in enumerate(zip(array.shape, shapes[i])):
144 if not j and not check_batch_axis:

ValueError: Error when checking input: expected input_1 to have 4 dimensions, but got array with shape (8, 1)

Hi!
I'm trying to get into object detection. I've tried to use your SSD7 model on Udacity dataset to detect traffic lights. I've managed to sometimes to spot some traffic lights, but nothing more. It wasn't doing well, so I tried more simple example.

Now I'm trying to use it for one of the simplest tasks - recognizing digits.
I'm generating artificial dataset using this notebook.
I'm training in this notebook using your ssd7 model. But as you can see in the pic in the bottom of the notebook, it is not doing well. I don't understand why it not working at all.

Can you please point me what am I doing wrong?
Thanks, Alexander.

Hi, can u give me the model that have trained which contains the weights data, i want to test quickly.thanks u

Hi,

First of all, really great work with the documentation. It was very helpful.

I have been having issue when working with a custom dataset I made of various image sizes, I keep running into an issue when it comes to encoding the bouding boxes at line: y_encoded[i,assign_indices,:-8] = np.concatenate((class_vector[int(true_box[0])], true_box[1:]), axis=0)

It raises an IndexError: index 14 is out of bounds for axis 0 with size 14.

Any ideas?

Thank you very much for your code and detailed explanation! It's really helpful.
I used ssd300 to train other dataset, and then used the trained model to do prediction. But I got the following output:

How should I set the config to get only 1 bounding box output, please? Thank you very much!

Hi
Thanks for your great work
I am trying to understand, how to use it,
it would have been useful to have a sample of some image and there corresponding csv file with labels, so can reproduce the format.

I have tried the following format
image_name,classID,xmin,ymin,xmax,ymax
xmin,ymin,xmax,ymax - are the in pixels position's on the image
Checked them with drawing on image and they are correct.

air1.jpg, 1, 352, 211, 445, 300
air2.jpg, 1, 408, 248, 489, 334
air3.jpg, 1, 339, 194, 460, 296
air4.jpg, 1, 192, 295, 359, 408
air4.jpg, 1, 311, 177, 443, 262
air4.jpg, 1, 312, 430, 438, 509
air4.jpg, 1, 445, 492, 617, 547
air4.jpg, 1, 446, 290, 606, 391
air4.jpg, 1, 457, 59, 601, 171
air4.jpg, 1, 572, 405, 698, 478
air4.jpg, 1, 578, 176, 704, 254
air4.jpg, 1, 670, 281, 818, 386
air5.jpg, 1, 295, 219, 444, 301
air6.jpg, 1, 50, 57, 734, 321
air6.jpg, 1, 54, 862, 1082, 1217
air6.jpg, 1, 70, 1333, 809, 1622
air6.jpg, 1, 72, 1688, 951, 1946
air6.jpg, 1, 92, 465, 951, 783
air6.jpg, 1, 871, 92, 1697, 330
air6.jpg, 1, 895, 1272, 1796, 1593
air6.jpg, 1, 980, 1636, 1933, 2012
air6.jpg, 1, 1028, 439, 1760, 685
air6.jpg, 1, 1169, 743, 1750, 1271
air6.jpg, 1, 1809, 867, 2659, 1188
air6.jpg, 1, 1823, 491, 2728, 818
air6.jpg, 1, 1838, 46, 2601, 445
air6.jpg, 1, 1839, 1286, 2731, 1561
air6.jpg, 1, 1966, 1633, 2696, 1925
bird1.jpg, 2, 591, 210, 789, 366
bird2.jpg, 2, 907, 309, 1213, 520
bird2.jpg, 2, 1270, 572, 1724, 715
bird3.jpg, 2, 856, 695, 1168, 838
bird3.jpg, 2, 873, 224, 1080, 387
bird3.jpg, 2, 924, 454, 1080, 562
bird4.jpg, 2, 1452, 1008, 2628, 1594

Used your parse_csv() function in ssd_batch_generator.py

for testing I created val and train with the same image list
then continued for training using the ssd_300 training Jupiter notebook

1: Instantiate to BatchGenerator objects: One for training, one for validation.

train_dataset = BatchGenerator(box_output_format=['class_id', 'xmin', 'ymin', 'xmax', 'ymax'])
val_dataset = BatchGenerator(box_output_format=['class_id', 'xmin', 'ymin', 'xmax', 'ymax'])

train_dataset.parse_csv(images_path = images_path,
labels_path = labesl_path,
input_format=['image_name', 'class_id','xmin', 'xmax', 'ymin', 'ymax'],
include_classes = 'all',
random_sample = False,
ret = True)
val_dataset.parse_csv(images_path = images_path,
labels_path = labesl_path,
input_format=['image_name', 'class_id','xmin', 'xmax', 'ymin', 'ymax'],
include_classes = 'all',
random_sample = False,
ret = True)

but learning fails

Epoch 1/10
Epoch 00000: val_loss improved from inf to 210993.18750, saving model to ssd300_weights_epoch-00_loss-28.2251_val_loss-210993.1875.h5
1/1 [==============================] - 73s - loss: 28.2251 - val_loss: 210993.1875
Epoch 2/10
Epoch 00001: val_loss did not improve
1/1 [==============================] - 68s - loss: 238422.5000 - val_loss: 134127960.0000
Epoch 3/10
Epoch 00002: val_loss did not improve
1/1 [==============================] - 77s - loss: 176149040.0000 - val_loss: 4924609024.0000
Epoch 4/10
Epoch 00003: val_loss did not improve
1/1 [==============================] - 74s - loss: 2190408704.0000 - val_loss: 213619638272.0000

Can you please help me understand where is the fault?
did I set the labels format correct in the csv file?
did I miss something else?

Hi Pierluigi,

First, thanks for brilliant well documented implementation of ssd! :)

It seems that there is a little yet annoying bug in image height/width processing. By default you're using 300x300 and it works fine. However, if you set w/h to any uneven values (say 1260x720) then it fails to run.

Cheers
Kirill

Hello!

Thank you for all the work you've done to port SSD to Keras!

I'm trying to train a subset of annotated data from MS COCO using ssd7_training.ipynb with two classes (background and the one feature I'm interested in) and I've largely left your parameters the way they are in the repo except for file and directory paths. I've made it to 4. Run the training without a hitch (n_train_samples and n_val_samples are giving the correct number of images I have in my directories).

Here is a sample of my training label data I've tried to mimic the labels.csv example in your Udacity training dataset:

Thinking my data is set up correctly, I try to train it by executing what is included in ssd7_training.ipynb:

# TODO: Set the number of epochs to train for.
epochs = 10

history = model.fit_generator(generator = train_generator,
                              steps_per_epoch = ceil(n_train_samples/batch_size),
                              epochs = epochs,
                              callbacks = [ModelCheckpoint('ssd7_weights_epoch-{epoch:02d}_loss-{loss:.4f}.h5',
                                                           monitor='val_loss',
                                                           verbose=1,
                                                           save_best_only=True,
                                                           save_weights_only=True,
                                                           mode='auto',
                                                           period=1),
                                           EarlyStopping(monitor='val_loss',
                                                         min_delta=0.001,
                                                         patience=2),
                                           ReduceLROnPlateau(monitor='val_loss',
                                                             factor=0.5,
                                                             patience=0,
                                                             epsilon=0.001,
                                                             cooldown=0)],
                              validation_data = val_generator,
                              validation_steps = ceil(n_val_samples/batch_size))

# TODO: Set the filename (without the .h5 file extension!) under which to save the model and weights.
#       Do the same in the `ModelCheckpoint` callback above.
model_name = 'ssd7'
model.save('{}.h5'.format(model_name))
model.save_weights('{}_weights.h5'.format(model_name))

print()
print("Model saved under {}.h5".format(model_name))
print("Weights also saved separately under {}_weights.h5".format(model_name))
print()

But I'm experiencing the following error when I start training:

Epoch 1/10
Exception in thread Thread-6:
Traceback (most recent call last):
  File "/home/roman/.conda/envs/deep-learning/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/home/roman/.conda/envs/deep-learning/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/home/roman/.conda/envs/deep-learning/lib/python3.6/site-packages/keras/utils/data_utils.py", line 568, in data_generator_task
    generator_output = next(self._generator)
  File "/home/roman/git/ssd_keras/ssd_batch_generator.py", line 984, in generate
    y_true = ssd_box_encoder.encode_y(batch_y, diagnostics) # Encode the labels into the `y_true` tensor that the cost function needs
  File "/home/roman/git/ssd_keras/ssd_box_encode_decode_utils.py", line 943, in encode_y
    y_encoded[i,best_match_index,:-8] = np.concatenate((class_vector[int(true_box[0])], true_box[1:]), axis=0) # Write the ground truth box coordinates and class to the best match anchor box position
IndexError: index 1 is out of bounds for axis 0 with size 1

---------------------------------------------------------------------
StopIteration                       Traceback (most recent call last)
<ipython-input-24-3f284dc3f325> in <module>()
     21                                                              cooldown=0)],
     22                               validation_data = val_generator,
---> 23                               validation_steps = ceil(n_val_samples/batch_size))
     24 
     25 # TODO: Set the filename (without the .h5 file extension!) under which to save the model and weights.

~/.conda/envs/deep-learning/lib/python3.6/site-packages/keras/legacy/interfaces.py in wrapper(*args, **kwargs)
     85                 warnings.warn('Update your `' + object_name +
     86                               '` call to the Keras 2 API: ' + signature, stacklevel=2)
---> 87             return func(*args, **kwargs)
     88         wrapper._original_function = func
     89         return wrapper

~/.conda/envs/deep-learning/lib/python3.6/site-packages/keras/engine/training.py in fit_generator(self, generator, steps_per_epoch, epochs, verbose, callbacks, validation_data, validation_steps, class_weight, max_queue_size, workers, use_multiprocessing, shuffle, initial_epoch)
   2009                 batch_index = 0
   2010                 while steps_done < steps_per_epoch:
-> 2011                     generator_output = next(output_generator)
   2012 
   2013                     if not hasattr(generator_output, '__len__'):

StopIteration: 

Is this error indicative of something that I haven't correctly set up? I'm using Keras 2.0.8. I'm on commit 65c5ca572ec

I use train_ssd7 to train 67 pictures of udacity dataset provided by you, but it seems can not work likehere.

There are something I have changed:
(1) dataset: only 67 pictures as my dataset here
(2) learning rate: 0.00001, because 0.01 and 0.0001 keep loss staying large, I reduce learning rate to hope it can learn something at least, but it turns out the same bad result
(3) epoch: 500, because I have little dataset, I think I need to let it train the times of the number of pictures according to the parameter in ssd7
(4) EarlyStopping: I use patience=epochs to turn off it, because I think small dataset can not learn much in each epoch, it needs more epochs to simulate large dataset

I think small dataset would cause over-fitting which can recognize the object provided by my dataset but can not tell objects from other source, however, it seems that it does not learn anything.
Am I wrong?
Thank you!!

I convert the train_ssd300.ipnb into .py file, and execute it to train on PASCAL VOC 2007, 2012, some errors were thrown as below:
os environment:
ubuntu 16.04
tf: tensorflow-gpu 1.30
keras: 2.0

history = model.fit_generator(generator = train_generator,
steps_per_epoch = ceil(n_train_samples/batch_size),
epochs = epochs,
callbacks = [ModelCheckpoint('ssd300_weights_epoch-{epoch:02d}_loss-{loss:.4f}_val_loss-{val_loss:.4f}.h5',
monitor='val_loss',
verbose=1,
save_best_only=True,
save_weights_only=True,
mode='auto',
period=1),
LearningRateScheduler(lr_schedule),
EarlyStopping(monitor='val_loss',
min_delta=0.001,
patience=2)],
validation_data = val_generator,
validation_steps = ceil(n_val_samples/batch_size))

File "train_ssd300.py", line 275, in
validation_steps = ceil(n_val_samples/batch_size))
File "/home/junhao.li/anaconda2/envs/py35/lib/python3.5/site-packages/keras/legacy/interfaces.py", line 87, in wrapper
return func(*args, **kwargs)
File "/home/junhao.li/anaconda2/envs/py35/lib/python3.5/site-packages/keras/engine/training.py", line 2016, in fit_generator
self._make_train_function()
File "/home/junhao.li/anaconda2/envs/py35/lib/python3.5/site-packages/keras/engine/training.py", line 990, in _make_train_function
loss=self.total_loss)
File "/home/junhao.li/anaconda2/envs/py35/lib/python3.5/site-packages/keras/legacy/interfaces.py", line 87, in wrapper
return func(*args, **kwargs)
File "/home/junhao.li/anaconda2/envs/py35/lib/python3.5/site-packages/keras/optimizers.py", line 424, in get_updates
lr_t = lr * (K.sqrt(1. - K.pow(self.beta_2, t)) /
File "/home/junhao.li/anaconda2/envs/py35/lib/python3.5/site-packages/keras/backend/tensorflow_backend.py", line 1443, in sqrt
x = tf.clip_by_value(x, zero, inf)
File "/home/junhao.li/anaconda2/envs/py35/lib/python3.5/site-packages/tensorflow/python/ops/clip_ops.py", line 58, in clip_by_value
[t, clip_value_min, clip_value_max]) as name:
File "/home/junhao.li/anaconda2/envs/py35/lib/python3.5/contextlib.py", line 59, in enter
return next(self.gen)
File "/home/junhao.li/anaconda2/envs/py35/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 4521, in name_scope
g = _get_graph_from_inputs(values)
File "/home/junhao.li/anaconda2/envs/py35/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 4262, in _get_graph_from_inputs
_assert_same_graph(original_graph_element, graph_element)
File "/home/junhao.li/anaconda2/envs/py35/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 4201, in _assert_same_graph
"%s must be from the same graph as %s." % (item, original_item))
ValueError: Tensor("training/Adam/Const:0", shape=(), dtype=float32) must be from the same graph as Tensor("sub:0", shape=(), dtype=float32

Thanks a lot for great materials here :) I would like to make prediction on a single image (e.g. 000.jpg) NOT included in the validation set. Can anyone help me to revise the code here (train_ssd7.ipynb) for the purpose? I really appreciate your kind attention and warm help!

@pierluigiferrari HI,greate work here.

I want to use this project to train on a small image set for a experiment.
I got a image set which has been organized in KITTI data format.I don't know how to train it.Could you please give a simple instructions about training?
Thank you !

whenever i run training i get this error after a few steps

File "/home/saghir/keras/ssd_keras/ssd_batch_generator.py", line 1177, in generate
batch_inverse_coord_transform.pop(j)
AttributeError: 'numpy.ndarray' object has no attribute 'pop'

anyone know how to solve this

Hey I think I found a bug in ssd_batch_geneartor.parse_csv. The issue comes from an example file as so:

fname, xmin, xmax, ymin,ymax,class_id
a.jpg,1,2,3,4,1,
b.jpg,1,2,3,4,1,
c.jpg,1,2,3,4,1

The last c.jpg will not be added to filenames and labels. It is skipped.

Notice that the last row is a new file (important because your logic looks at this condition).

The error appears here.

                else: # If this box belongs to a new image file
                    self.labels.append(np.stack(current_labels, axis=0)) # appending the PREVIOUS labels
                    self.filenames.append(os.path.join(self.images_path, current_file)) # appending the PREVIOUS filenames
                    current_labels = []
                    current_file = i[0] # This will never be added, we are on the last line/iteration of loop
                    current_labels.append(i[1:]) # This will never be added, we are on the last line/iteration of loop

Proposal to fix (which I'm using and worked okay:

                else: # If this box belongs to a new image file
                    self.labels.append(np.stack(current_labels, axis=0))
                    self.filenames.append(os.path.join(self.images_path, current_file))
                    current_labels = []
                    current_file = i[0]
                    current_labels.append(i[1:])
                    if idx == len(data)-1: # If this is the last line of the CSV file
                        self.labels.append(np.stack(current_labels, axis=0))
                        self.filenames.append(os.path.join(self.images_path, current_file))

Hope it helps!

Hi,

I got to train a model with a custom dataset, though I have run into an issue where the model will sometimes not make any predictions for a logo - when there is one in an image. I decided to measure this and called it "hit_rate", with a hit being making a prediction regardless if its correct, and on my dataset (test data), its quite low - 13.48% of the images.

At the moment I have just kept it aside to work on other things while I think about what the potential problems could be. I wanted to ask if you have experienced something similar.

My suspicion (ordered by which one I think is most probable):

• I might not have enough ground truth boxes for the training (i.e. not much matches), which means it didn't get to train on much data.
• Varying image sizes of the wild in the training data means its harder for it to learn to predictions of bounding boxes, and as a result that limitation is affecting the predictions.
• Something to do with the architecture needing fine-tuning (though I dont think this is the case as I see the SSDModel doesnt seem to have a problem of objects having varying sizes)

Thanks in advance.

should the model in ssd_300

conv8_1 = Conv2D(128, (1, 1), activation='relu', padding='same', kernel_initializer='he_normal', name='conv8_1')(conv7_2)
    conv8_2 = Conv2D(256, (3, 3), strides=(1, 1), activation='relu', padding='valid', kernel_initializer='he_normal', name='conv8_2')(conv8_1)

    conv9_1 = Conv2D(128, (1, 1), activation='relu', padding='same', kernel_initializer='he_normal', name='conv9_1')(conv8_2)
    conv9_2 = Conv2D(256, (3, 3), strides=(1, 1), activation='relu', padding='valid', kernel_initializer='he_normal', name='conv9_2')(conv9_1)

the strides be (3,3) not (1,1) from original paper?

Hello,

When running ssd300_training.ipynb, I get the error:local variable 'intersection' referenced before assignment. This is caused by the if condition in line 66 in ssd_box_encode_decode_utils.py which doesn't handle the third (and default) case: centroids.

Thanks

This is not an issue but a question. First of all thank you very much for creating and maintaining this.

I am looking to re-train SSD on COCO-text and I am new to deep learning.

Should I start with ssd7_training.ipynb and then see if I get decent results on the test set? If so, then train on ssd300_training.ipynb? Are there any pointers you'd recommend before I begin the training?

Hi -
Great example! I am running through training steps on SSD300 and Pascal VOC data. Made it through all 10 epochs during the training, saving weights files for every epoch. On the tenth epoch, a crash occurs (even though it looks like the file was saved). Is this a bounds issue? I will be retrying a different number of epochs to see if this only happens on the last epoch.
Thanks,
Rob

UnicodeDecodeError Traceback (most recent call last)
in ()
23
24 model_name = 'ssd300_0'
---> 25 model.save('./{}.h5'.format(model_name))
26 model.save_weights('./{}_weights.h5'.format(model_name))
27

~\Anaconda3\envs\tensorflow\lib\site-packages\keras\engine\topology.py in save(self, filepath, overwrite, include_optimizer)
2554 """
2555 from ..models import save_model
-> 2556 save_model(self, filepath, overwrite, include_optimizer)
2557
2558 def save_weights(self, filepath, overwrite=True):

~\Anaconda3\envs\tensorflow\lib\site-packages\keras\models.py in save_model(model, filepath, overwrite, include_optimizer)
105 f.attrs['model_config'] = json.dumps({
106 'class_name': model.class.name,
--> 107 'config': model.get_config()
108 }, default=get_json_type).encode('utf8')
109

~\Anaconda3\envs\tensorflow\lib\site-packages\keras\engine\topology.py in get_config(self)
2327 for layer in self.layers: # From the earliest layers on.
2328 layer_class_name = layer.class.name
-> 2329 layer_config = layer.get_config()
2330 filtered_inbound_nodes = []
2331 for original_node_index, node in enumerate(layer.inbound_nodes):

~\Anaconda3\envs\tensorflow\lib\site-packages\keras\layers\core.py in get_config(self)
658 def get_config(self):
659 if isinstance(self.function, python_types.LambdaType):
--> 660 function = func_dump(self.function)
661 function_type = 'lambda'
662 else:

~\Anaconda3\envs\tensorflow\lib\site-packages\keras\utils\generic_utils.py in func_dump(func)
173 A tuple (code, defaults, closure).
174 """
--> 175 code = marshal.dumps(func.code).decode('raw_unicode_escape')
176 defaults = func.defaults
177 if func.closure:

UnicodeDecodeError: 'rawunicodeescape' codec can't decode bytes in position 80-81: truncated \uXXXX

Hi Pierluigi, thank you very much for your pretrained model.

I tried to load 'VGG_VOC0712Plus_SSD_300x300_ft_iter_160000.h5', and this predicted well. But when I loaded 'VGG_ILSVRC2016_SSD_300x300_iter_440000.h5', there were some errors. After I changed the 'n_classes' from 21 to 201, it can be loaded successfully, but there are still errors when I use it for prediction, and I think it is because of the 'classes'.

So did you use all the classes when you trained the model with imagenet? Or you just trained it with 200 classes of them? How should I modify the code if I want to use this pretrained model?

Thank you very much!

In keras_ssd_loss.py , in compute_loss,
required shape of y_true and y_pred is [classes one-hot encoded, 4 ground truth box coordinates, 4 arbitrary entries], latter 4 for anchor box coordinates

But in model fun in keras_ssd7.py,
Output shape of y_pred or predictions is (batch, n_boxes_total, n_classes + 4 + 8) where in addition to the above, there are 4 more terms for variances.

But in-spite of this contradiction that y_pred(from the model) having different shape than y_true, the model still works. I am confused by this.
If you could tell me how it works, in order to facilitate the understanding of the project, I would be ever grateful.

Thanks a lot for great materials here :) Retraining the SSD7 model is done well before I close the Keras program for the first time. However, I get an error message below after I restart the Keras program and retrain the SSD7 model (I run 1., skip 2.1 then I run 2.2 and the rest in train_ssd7.ipynb; img_height = 260, img_width = 200, img_channels = 3). Can anyone help to retrain the SSD7 model after reopening the Keras program? I really appreciate your kind attention and warm help!

#Error message:
ValueError: Error when checking target: expected predictions to have shape (None, 3424, 16) but got array with shape (30, 10316, 16)

Hi,

Didn't know how to tell you this so I opened an issue: I have translated the SSD7 materials into R using R's Keras wrapper. I of course cited you as the author wherever I could.

This is the repo: https://github.com/gsimchoni/ssdkeras

This is a blog post explaining how I did Snapchat-like face-masks with the SSD7 on faces: http://giorasimchoni.com/2017/12/04/2017-12-04-snap-fu/

Tell me if you'd like to change anything in citations, notice the package is under construction.

Thanks Pierluigi!

Hi，I want to use ssd7 to train my own dataset,but my images are 12001600,I wonder whether I resize images to 300300 or change the network to fit my dataset.

Hi pierluigiferrari, thank you very much. after I study "train_ssd7", there are some process I am confused, hope you can help me. In "keras_ssd7.py", do we use both "only boxes" and "anchor_box"( with "variance box") to learn localization? what I learn from your code is the process as described below.
We use anchor_box to do SSDEncoder to match each layer from ground truth. But in "ssd_box_encode_decode_utils.py"->"def encode_y"->"y_encoded" which we use "y_encode_template" to save the match box for "y_encoder", however, (1)I do not find "y_encode_template" any change in "def encode_y".
After getting y_true from generator_anchor_boxes rather than anchor box, we compute loss for localization with only boxes without the others. (2)Does this mean what we actually train is "only boxes"? And I have no idea we use "anchor box" and "variance box" for what
(3)Besides, I really have no idea what the variance box function is. If there are all 1 value or [0.1 0.3 0.5 0.7], what does it really mean?
(4)And the last one I do not understand is why do we need 4+4+4 boxes to predict localization instead only 4 coordinate box to learn for that? Is there any augmentation for that?
I'm sorry about I am confused about your hard work on explanation.
But I learn a lot from your great work, thank you!

Hi everyone!

Nice job you have done! I've tested it and it's working just fine.

Did you ever try to export the generated model (h5 format) to tensorflow (pb format) and run it in a Android app?

I'was thinking to do that but no idea where I should start from.

Any suggestion would be great,

Flavio

Hi pierluigiferrai
Great work. Thanks for posting code.
I want to test the trained model on video data. How do I pass the video frames into Batch Generator?
I am waiting for your valuable suggestions.

Hi, thanks a lot for sharing your illuminating work but I have one issue tickles my mind.

When I try to train the model in my laptop, whic does not have good GPU memory, I lowered the batch size to 4 and then I began to see loss values around like 2 during training. Though it gets lower during training, it was a little bit suprise to see values bigger than 1.

What may cause this behaviour ? Should I take a look at 'SSDLoss'

Kind Regards

Error "ValueError: need at least one array to concatenate" has raised in file "ssd_box_encode_decode_utils.py", line 367, in decode_y when there is no detected object.

Hi,
I am trying to get better understanding of ssd object detection and your code is really helpful. Thanks

However, I found a difference between your ssd300 model and the paper in the model:
In particular I think that layers: conv6_1, conv6_2, conv7_1, conv7_2 are not included in the original paper (fig 2)
In the paper fc7 is connected to conv8 layers
I also think that padding in pool3 should be changed to same, so that the ouput shape of conv4 is 19x19 as in the paper

Thanks

How do I make the ssd_keras package?/module? work? For example sudo pip install ssd_keras is not functional.

Simple cloning or downloading the repository and then using it as proposed in the Readme does not work as well when I would like to use import module function as described for example in the train_ssd300.ipynb.

from keras_ssd300 import ssd_300
from keras_ssd_loss import SSDLoss
from keras_layer_AnchorBoxes import AnchorBoxes
from keras_layer_L2Normalization import L2Normalization
from ssd_box_encode_decode_utils import SSDBoxEncoder, decode_y, decode_y2
from ssd_batch_generator import BatchGenerator

The error occurs: ImportError: No module named keras_ssd300 (same for further modules)

Thanks for helping a beginner and sorry if the question might be stupid but my experience using python is, so far, very limited.

P.S.: I would like to follow the steps explained here and I would like to use my own model.

In the image below you can see the green ground truth bbox and the blue inferred bbox my the model. The issue is that this was after the model was loaded back into memory from disk via load_weights. If I do inference right after training in a notebook, the bbox is correctly sized from inference. Have you seen anything like this?

Hi, thanks for this wonderful work. I want test on your ssd7 dataset, I just downloaded the images file from your google drive, but how to get the train_label.csv file? Any link or resource would be very appreciated!!

I use pasocal voc2007 to train the model,and the loss is always is low,like0.2,Is anything wrong?

Hi there,

thanks for this repo!

When training, I get this error message:

Epoch 1/10
 44/517 [=>............................] - ETA: 8:18 - loss: 43.7727

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-8-fef4cc6cd229> in <module>()
     17                                                          patience=2)],
     18                               validation_data = val_generator,
---> 19                               validation_steps = ceil(n_val_samples/batch_size))
     20 
     21 # TODO: Set the filename (without the .h5 file extension!) under which to save the model and weights.

/usr/local/lib/python2.7/dist-packages/keras/legacy/interfaces.pyc in wrapper(*args, **kwargs)
     85                 warnings.warn('Update your `' + object_name +
     86                               '` call to the Keras 2 API: ' + signature, stacklevel=2)
---> 87             return func(*args, **kwargs)
     88         wrapper._original_function = func
     89         return wrapper

/usr/local/lib/python2.7/dist-packages/keras/engine/training.pyc in fit_generator(self, generator, steps_per_epoch, epochs, verbose, callbacks, validation_data, validation_steps, class_weight, max_queue_size, workers, use_multiprocessing, shuffle, initial_epoch)
   2113                 batch_index = 0
   2114                 while steps_done < steps_per_epoch:
-> 2115                     generator_output = next(output_generator)
   2116 
   2117                     if not hasattr(generator_output, '__len__'):

/usr/local/lib/python2.7/dist-packages/keras/utils/data_utils.pyc in get(self)
    733             success, value = self.queue.get()
    734             if not success:
--> 735                 six.reraise(value.__class__, value, value.__traceback__)

/usr/local/lib/python2.7/dist-packages/keras/utils/data_utils.pyc in data_generator_task()
    633                 try:
    634                     if self._use_multiprocessing or self.queue.qsize() < max_queue_size:
--> 635                         generator_output = next(self._generator)
    636                         self.queue.put((True, generator_output))
    637                     else:

/home/ubuntu/ssd_keras/ssd_batch_generator.pyc in generate(self, batch_size, shuffle, train, ssd_box_encoder, equalize, brightness, flip, translate, scale, max_crop_and_resize, full_crop_and_resize, random_crop, crop, resize, gray, limit_boxes, include_thresh, subtract_mean, divide_by_stddev, swap_channels, diagnostics)
    962             for j in sorted(batch_items_to_remove, reverse=True):
    963                 batch_X.pop(j)
--> 964                 batch_y.pop(j) # This isn't efficient, but it hopefully should not need to be done often anyway.
    965 
    966             # CAUTION: Converting `batch_X` into an array will result in an empty batch if the images have varying sizes.

AttributeError: 'numpy.ndarray' object has no attribute 'pop'

Please, what is the reason for this?
''pop'' can normally be used for lists, as far as I know, not for ndarrays... is this a bug in the code or something else?
Thanks!

I'm presenting this as a suggestion, because skewing RGB channels individually may not produce the desired effects to improve contrast.

By the way, how to transfer the caffemodel to .hdf5 file?

hello pierluigiferrari, thanks for your works
the explanation of code is so clear

however, when I test train_ssd7.ipynb, I can't train such great result as yours
it seems to learn for nothing

I only modify 2 places of your code:
(1) init() of BatchGenerator in ssd_batch_generator.py
because it seems to need some parameters in train_ssd7.ipynb to work
(2)the train and val data path in train_ssd7.ipynb
I download the data from your dataset, udacity_driving_datasets
and set both of train and val data path to the same path

and here is my result after modification
I don't know why it can't work
hope you can help me
thank you!

Thanks for the amazing repository and explanation.
I am trying to learn a model with one class and 1000 examples, using code very similar to train_ssd7.ipynb
But the model is always starting from loss=0 and doesn't learn anything - what may be the cause of that ?