I would like to visualise the first Convolutional Layer and its weights, and to do so I have written the below code.

    get_layer_output = K.function([model.layers[0].input], [model.layers[1].get_output(train=False)])
    layer_output = get_layer_output([test_data])[0]

    print layer_output.shape

This returns a numpy array of shape (6, 53, 8, 28, 28) - of which I am unable to plot using matplotlib's imshow.

Can anyone suggest a method of plotting the output of the first ConvLayer on a sequence of inputs for a single example?

Thank you,

Keiron.

Hey there,

I have been playing around with your keras-extra add on for awhile - but I have been unable to yield any decent results whilst using it as opposed to my standard libraries.

Below is my code:

def create_lcrn(number_of_timesteps, height, width):
    print "Creating the LRCN:"
    model = Sequential()

    # Convolutional Layers
    model.add(TimeDistributedConvolution2D(32, 3, 3, border_mode='same', input_shape=(number_of_timesteps, 1, height, width)))
    model.add(Activation('relu'))

    model.add(TimeDistributedMaxPooling2D(pool_size=(3, 3)))
    model.add(Dropout(0.25))

    model.add(TimeDistributedConvolution2D(64, 3, 3, border_mode='same'))
    model.add(Activation('relu'))

    model.add(TimeDistributedMaxPooling2D(pool_size=(3, 3)))
    model.add(Dropout(0.25))

    model.add(TimeDistributedFlatten())

    model.add(LSTM(128, return_sequences=True))
    model.add(Dropout(0.2))
    model.add(LSTM(64))

    model.add(Dense(2))
    model.add(Activation('softmax'))

    # o = SGD(lr=0.1, decay=1e-6, momentum=0.9, nesterov=True)
    o = RMSprop(lr=0.0005, rho=0.9, epsilon=1e-06)
    model.compile(loss="binary_crossentropy", optimizer=o)
    print "\t Model Created!"
    return model

def train_model(model, train_data, train_label):
    train_label = np_utils.to_categorical(train_label, 2)
    model.fit(train_data, train_label, batch_size=2, nb_epoch=10, verbose=1, validation_split=0.2, show_accuracy=True)

if __name__ == "__main__":
    # Details for training
    number_of_folds = 2

    # Details for data
    number_of_timesteps, height, width = 53, 60, 80

    all_test_labels = []
    all_prediction_probabilities = []
    all_predictions = []


    data, labels = load_data(height, width, number_of_timesteps)
    number_of_samples = data.shape[0]

    print "\tVideo loaded in shape of:", data.shape
    print "\t\tNumber of samples:", number_of_samples, ", Number of timesteps:", number_of_timesteps


    stratified_k_fold = StratifiedKFold(labels, n_folds=number_of_folds, shuffle=True)
    for i, (train, test) in enumerate(stratified_k_fold):
        model = None
        model = create_lcrn(number_of_timesteps, height, width)
        train_model(model, data[train], labels[train])

And here is the output.

Using Theano backend.
Loading PoD video data:
    Video loaded in shape of: (52, 53, 1, 60, 80)
        Number of samples: 52 , Number of timesteps: 53
Creating the LRCN:
     Model Created!
Train on 20 samples, validate on 6 samples
Epoch 1/10
20/20 [==============================] - 55s - loss: 0.4085 - acc: 0.8000 - val_loss: 2.5858 - val_acc: 0.0000
Epoch 2/10
20/20 [==============================] - 55s - loss: 0.2131 - acc: 0.9500 - val_loss: 2.9949 - val_acc: 0.0000
Epoch 3/10
20/20 [==============================] - 55s - loss: 0.2075 - acc: 0.9500 - val_loss: 3.0842 - val_acc: 0.0000

Any help would be gratefully appreciated.

Is there example code anywhere? This is an exciting addition to keras, and I'd love to see it in action. I attempted to adapt your 'untested' code snippet from

keras-team/keras#401

but had trouble going from the TimeDistributedFlatten to the LSTM layers.

model = Sequential()
model.add(TimeDistributedConvolution2D(8, 4, 4, border_mode='same', input_shape=(n_timesteps,1, 28, 28)))
model.add(TimeDistributedMaxPooling2D(pool_size=(2, 2)))
model.add(Activation('relu'))
model.add(TimeDistributedFlatten())
model.add(LSTM(256, return_sequences=False))
model.add(Dense(nb_classes))
model.add(Activation('softmax')

That yields the following dimension mismatch:
Inputs shapes: [(32, 392), (1568, 256)]

My test case is simply processing windows within MNIST images sequentially. My window size is (15,15) resulting in 196 windows.

The shape of X_train is (60000, 196, 1, 15, 15)

If I instead try return_sequences = True in the RNN layer, I get
AssertionError: Incompatible shapes: layer expected input with ndim=2 but previous layer has output_shape (None, 196, 256)

Is there an example anywhere that I could use to troubleshoot?

Thanks

Hi, @anayebi .
I'm trying to use your 'keras-extra' to build a lrcn network.
I installed 'keras-extra' following your README instruction.
Unfortunately, It returns an error message as below.

from keras.layers.extra import TimeDistributedConvolution2D
ImportError: No module named extra

A "extra.py" file is already in 'keras/layers/' folder and auto complete function properly works
when writing the "from keras.layers.extra import layername" in Pycharm.

Could you give a suggestion to solve it?
Thx!!

Hey man,

Is this supported in Keras 3.1? I get the following error:

`Exception Traceback (most recent call last)
/usr/lib/python2.7/dist-packages/IPython/utils/py3compat.pyc in execfile(fname, *where)
202 else:
203 filename = fname
--> 204 builtin.execfile(filename, *where)

/home/ubuntumax/keras_branch/keras-dsb/experiment.py in ()
304 create(sys.argv[2])
305 elif sys.argv[1] == 'run':
--> 306 run(sys.argv[2])
307 elif sys.argv[1] == 'continue':
308 run(sys.argv[2], cont=True)

/home/ubuntumax/keras_branch/keras-dsb/experiment.py in run(experiment_name, cont)
59 print('Loading models...')
60 model_module = import_module(experiment_path, MODEL_PY)
---> 61 model_systole = model_module.get_model()
62 model_diastole = model_module.get_model()
63 # save model to JSON

/home/ubuntumax/keras_branch/keras-dsb/experiments/VGG_Time_Distributed_LSTM/model.pyc in get_model()
20 def get_model():
21 conv = Sequential()
---> 22 conv.add(Activation(activation=scale, input_shape=(30, 1, 128, 128)))
23 conv.add(TimeDistributedConvolution2D(64, 3, 3, border_mode='same'))
24 conv.add(Activation('relu'))

/usr/local/lib/python2.7/dist-packages/keras/layers/core.pyc in init(self, activation, **kwargs)
652 '''
653 def init(self, activation, **kwargs):
--> 654 super(Activation, self).init(**kwargs)
655 self.activation = activations.get(activation)
656

/usr/local/lib/python2.7/dist-packages/keras/layers/core.pyc in init(self, **kwargs)
41 assert kwarg in allowed_kwargs, 'Keyword argument not understood: ' + kwarg
42 if 'input_shape' in kwargs:
---> 43 self.set_input_shape((None,) + tuple(kwargs['input_shape']))
44 if 'batch_input_shape' in kwargs:
45 self.set_input_shape(tuple(kwargs['batch_input_shape']))

/usr/local/lib/python2.7/dist-packages/keras/layers/core.pyc in set_input_shape(self, input_shape)
138 ', was provided with input shape ' + str(input_shape))
139 self._input_shape = input_shape
--> 140 self.input = K.placeholder(shape=self._input_shape)
141 self.build()
142

/usr/local/lib/python2.7/dist-packages/keras/backend/theano_backend.pyc in placeholder(shape, ndim, dtype, name)
53 return T.tensor4(name=name, dtype=dtype)
54 else:
---> 55 raise Exception('ndim too large: ' + str(ndim))
56
57

Exception: ndim too large: 5
`

My actual code:

conv = Sequential() conv.add(Activation(activation=scale, input_shape=(30, 1, 128, 128))) conv.add(TimeDistributedConvolution2D(64, 3, 3, border_mode='same')) conv.add(Activation('relu')) # model.add(TimeDistributedConvolution2D(64, 3, 3)) # model.add(Activation('relu')) conv.add(TimeDistributedMaxPooling2D(pool_size=(2, 2))) conv.add(TimeDistributedConvolution2D(128, 3, 3, border_mode='same')) conv.add(Activation('relu')) # model.add(TimeDistributedConvolution2D(128, 3, 3)) # model.add(Activation('relu')) conv.add(TimeDistributedMaxPooling2D(pool_size=(2, 2))) conv.add(TimeDistributedConvolution2D(256, 3, 3, border_mode='same')) conv.add(Activation('relu')) # model.add(TimeDistributedConvolution2D(256, 3, 3)) # model.add(Activation('relu')) conv.add(TimeDistributedMaxPooling2D(pool_size=(2, 2))) conv.add(TimeDistributedConvolution2D(512, 3, 3, border_mode='same')) conv.add(Activation('relu')) # model.add(TimeDistributedConvolution2D(512, 3, 3)) # model.add(Activation('relu')) conv.add(TimeDistributedMaxPooling2D(pool_size=(2, 2))) conv.add(Activation('relu')) conv.add(TimeDistributedConvolution2D(512, 3, 3, border_mode='same')) conv.add(Activation('relu')) # model.add(TimeDistributedConvolution2D(512, 3, 3)) # model.add(Activation('relu')) conv.add(TimeDistributedMaxPooling2D(pool_size=(2, 2))) conv.add(Activation('relu')) conv.add(TimeDistributedConvolution2D(512, 3, 3, border_mode='same')) conv.add(Activation('relu')) # model.add(TimeDistributedConvolution2D(512, 3, 3)) # model.add(Activation('relu')) conv.add(TimeDistributedMaxPooling2D(pool_size=(2, 2))) conv.add(Activation('relu')) # model.add(TimeDistributedConvolution2D(512, 3, 3, border_mode='same')) # model.add(Activation('relu')) # # model.add(TimeDistributedConvolution2D(512, 3, 3)) # # model.add(Activation('relu')) # model.add(TimeDistributedMaxPooling2D(pool_size=(2, 2))) # model.add(Activation('relu')) # model.add(TimeDistributedConvolution2D(512, 3, 3, border_mode='same')) # model.add(Activation('relu')) # # model.add(TimeDistributedConvolution2D(512, 3, 3)) # # model.add(Activation('relu')) # model.add(TimeDistributedMaxPooling2D(pool_size=(2, 2))) # model.add(Activation('relu')) conv.add(TimeDistributedFlatten()) conv.add(LSTM(512, return_sequences=True)) conv.add(Dropout(0.5)) conv.add(Dense(1))

Hello anayebi,

I can't find the final file "setup.py" in the keras directory. I also don't know if I have to copy theano and tensorflow backend files.

Thanks,

Hatem

Hi, @anayebi , I am a newbie in keras, I want to do sequence classification (e.g., 2 classes, each element in the sequence is a 2d matrix or image) just like the first architecture in fig 3 in this article http://arxiv.org/pdf/1411.4389v4.pdf , the author said the label are determined by averaging the softmax of each time steps, this confuse me, can you tell me how to average the softmax of each time steps? and return sequences should to be set False or True?? Very thanks!
is my following code is right?

model = Sequential()
model.add(TimeDistributed(Convolution2D(8, 5, 5, border_mode='valid'), input_shape=(maxToAdd, 1, size1, size2)))
model.add(TimeDistributed(MaxPooling2D(pool_size=(2, 2),border_mode='valid')))
model.add(Activation('relu'))
model.add(TimeDistributed(Flatten()))
model.add(LSTM(output_dim=64, return_sequences=True))
model.add(Dropout(.5))
model.add(TimeDistributed(Dense(1)))
model.add(Activation('sigmoid'))
rmsprop = RMSprop()
model.compile(loss='mean_squared_error', optimizer=rmsprop, metrics=['accuracy'])

Hi,

I am trying to train LSTM, but while training accuracy remains zero in each epoch.
I have transformed data to multivariate Time-series data and also shape in three-dimensional shape.
I also have normalised data using minmaxsaller.

I have tried on a number of the epoch from 5 to 50 and batch size from 25 to 200.
I have tried data samples from 1000000 to 1000 but none is working.

Every time I am getting training accuracy zero only.

Can anyone help me in understanding it or suggest some more experiments.

Following is my network.

from keras.layers.core import Dense,Activation,Dropout
from keras.layers.recurrent import LSTM
from keras.models import Sequential
from keras.layers import Flatten

model = Sequential()
model.add(LSTM(50,return_sequences=True, input_shape=(X_train_values.shape[1], X_train_values.shape[2])))
model.add(Dropout(0.2))

model.add(Flatten())
model.add(Dense(1))

model.add(Activation('linear'))
model.compile(loss='mse',optimizer='rmsprop',metrics=['accuracy'])

history = model.fit(X_train_values, y_train.values,epochs=25, batch_size=30, verbose=2, shuffle=False)

n_hidden = 256
n_samples = 100
n_timesteps = 16

model = Sequential()
model.add(TimeDistributedConvolution2D(32, 5, 5, border_mode='same', input_shape=(n_timesteps, 28, 28)))
model.add(TimeDistributedMaxPooling2D(pool_size=(nb_pool, nb_pool)))
model.add(Activation('relu'))
model.add(TimeDistributedFlatten())
model.add(LSTM(256, return_sequences=True))
model.add(TimeDistributedDense(nb_classes))
model.add(Activation('softmax'))

the above code throws error as http://pastebin.com/E319wGeg

Hi,
I am trying to use conv2D plus LSTM to train a network with sequence of images (16 frames) and each sequence has 10 labels.

I made a simulation using Keras.extra and mnist data using the following code,

from future import absolute_import
from future import print_function
import numpy as np
import pandas as pd
from random import random
from sklearn.feature_extraction import image
from keras.preprocessing import sequence
from keras.optimizers import SGD, RMSprop, Adagrad
from keras.utils import np_utils
from keras.models import Sequential
from keras.initializations import normal, identity
from keras.layers.core import Dense, Dropout, Activation, Flatten, Permute, Reshape, TimeDistributedDense
from keras.layers.embeddings import Embedding
from keras.layers.recurrent import LSTM, GRU, SimpleRNN
from keras.layers.convolutional import Convolution1D, MaxPooling1D
from keras.layers.convolutional import Convolution2D, MaxPooling2D

from keras.layers.convolutional.TimeDistributedConvolution2D import TimeDistributedConvolution2

from keras.layers.extra import TimeDistributedConvolution2D, TimeDistributedMaxPooling2D, TimeDistributedFlatten
from keras.layers.recurrent import LSTM
from keras.datasets import mnist

(X_train, y_train), (X_test, y_test) = mnist.load_data()
X_train = X_train.astype('float32')
X_test = X_test.astype('float32')
'-----------------------------------------------Normalization------------------------------------------'
X_train /= 255
X_test /= 255
'-------------------------------------------------convert to the sequence of Images---------------------'
listy = [[] for i in range(16)]
listtr_ = [[] for i in range(16)]
tr_ = [[] for i in range(100)]
tr_lbl = [[] for i in range(100)]

for k in range(0,100,1):#number of samples
listy = [[] for i in range(16)]
#print(np.shape(listy))
index = 0
im = X_train[k]
lbl = y_train[k]
windowsize_r = 7
windowsize_c = 7
temp = []
# Crop out the window and calculate the histogram
for r in range(0,im.shape[0] , windowsize_r):
for c in range(0,im.shape[1] , windowsize_c):
window = im[r:r+windowsize_r,c:c+windowsize_c]
listy[index].append(window)
temp.append(window)
listtr_[index].append(lbl);
index = index + 1

tr_[k] = listy
tr_lbl[k] = listtr_

'-----------------------------------------------model parameters----------------------------------------'

lstm

n_hidden = 256
n_samples = 100
n_timesteps = 16

Convolution

filter_length = 3

number of convolutional filters to use

nb_filters = 32

size of pooling area for max pooling

nb_pool = 3

RNN

hidden_units = 100
nb_classes = 10
learning_rate = 0.1
batch_size = 20
nb_epochs = 100
nb_samples_tr = 100
nb_samples_te = 50
'------------------------------------------------------------------------------------------------------'
model = Sequential()
model.add(TimeDistributedConvolution2D(32, 3, 3, border_mode='same', input_shape=(n_timesteps,1, 7, 7)))
model.add(TimeDistributedMaxPooling2D(pool_size=(nb_pool, nb_pool)))
model.add(Activation('relu'))
model.add(TimeDistributedFlatten())
model.add(LSTM(256, return_sequences=True))
model.add(TimeDistributedDense(nb_classes))
model.add(Activation('softmax'))

rmsprop = RMSprop(lr=learning_rate)
model.compile(loss='categorical_crossentropy', optimizer=rmsprop)
tr_lbl = np.random.randn(100,10)
print('input shape:')
print(np.shape(tr_))
print('target shape:')
print(np.shape(tr_lbl))
model.fit(tr_, tr_lbl, batch_size=100, nb_epoch=10)
'----------------------------------------------------------------------------------------------

However, I still get stock on running the network properly. Here is the error I have received,

input shape:
(100, 16, 1, 7, 7)
target shape:
(100, 10)
Traceback (most recent call last):
File "myLSTM.py", line 169, in
model.fit(tr_, tr_lbl, batch_size=100, nb_epoch=10)
File "/usr/local/lib/python2.7/dist-packages/keras/models.py", line 503, in fit
raise Exception('All input arrays and the target array must '
Exception: All input arrays and the target array must have the same number of samples.

@anayebi

load model issue as follows. Any suggestions? Thanks

model = model_from_json(json_str)
File keras/models.py, line 166, in model_from_json
return model_from_config(config, custom_objects=custom_objects)
File "keras/models.py", line 177, in model_from_config
model = container_from_config(config, custom_objects=custom_objects)
File "keras/utils/layer_utils.py", line 57, in container_from_config
layer = container_from_config(layer_dict['nodes'].get(node['name']))
File "keras/utils/layer_utils.py", line 92, in container_from_config
base_layer = get_layer(name, layer_dict)
File "keras/utils/layer_utils.py", line 158, in get_layer
instantiate=True, kwargs=kwargs)
File "keras/utils/generic_utils.py", line 14, in get_from_module
str(identifier))
Exception: Invalid layer: TimeDistributedConvolution2D

When I use the layer method set_input_shape it sets the batch_input_shape, which in a convolutional layer case requires a 4dim tuple (batch_size,nb_channels,width,height).

When training a CNN with batch_input_shape rather than input_shape(3 dim), it throws an error at the end of the first epoch when training it.

This is the error:
Epoch 1/1 49920/50000 [============================>.] - ETA: 0s - loss: 1.9490 - acc: 0.3118Traceback (most recent call last): File "/usr/local/Cellar/python/2.7.10/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pdb.py", line 1314, in main pdb._runscript(mainpyfile) File "/usr/local/Cellar/python/2.7.10/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pdb.py", line 1233, in _runscript self.run(statement) File "/usr/local/Cellar/python/2.7.10/Frameworks/Python.framework/Versions/2.7/lib/python2.7/bdb.py", line 400, in run exec cmd in globals, locals File "<string>", line 1, in <module> File "CNN_CIFAR_10_6.py", line 6, in <module> from keras.datasets import cifar10 File "CNN_CIFAR_10_6.py", line 49, in trainingMethod2 modelsToGatherFV[k].fit(X_train_tmp, Y_training, batch_size=batch_size, nb_epoch=nb_epochs,show_accuracy=True) File "/usr/local/lib/python2.7/site-packages/keras/models.py", line 646, in fit shuffle=shuffle, metrics=metrics) File "/usr/local/lib/python2.7/site-packages/keras/models.py", line 280, in _fit outs = f(ins_batch) File "/usr/local/lib/python2.7/site-packages/keras/backend/theano_backend.py", line 384, in __call__ return self.function(*inputs) File "/usr/local/lib/python2.7/site-packages/theano/compile/function_module.py", line 871, in __call__ storage_map=getattr(self.fn, 'storage_map', None)) File "/usr/local/lib/python2.7/site-packages/theano/gof/link.py", line 314, in raise_with_op reraise(exc_type, exc_value, exc_trace) File "/usr/local/lib/python2.7/site-packages/theano/compile/function_module.py", line 859, in __call__ outputs = self.fn() ValueError: GpuReshape: cannot reshape input of shape (80, 32, 30, 30) to shape (128, 28800). Apply node that caused the error: GpuReshape{2}(GpuElemwise{Composite{(i0 * (i1 + Abs(i1)))},no_inplace}.0, TensorConstant{[ 128 28800]}) Toposort index: 48 Inputs types: [CudaNdarrayType(float32, 4D), TensorType(int64, vector)] Inputs shapes: [(80, 32, 30, 30), (2,)] Inputs strides: [(28800, 900, 30, 1), (8,)] Inputs values: ['not shown', array([ 128, 28800])] Outputs clients: [[GpuDimShuffle{1,0}(GpuReshape{2}.0), GpuDot22(GpuReshape{2}.0, <CudaNdarrayType(float32, matrix)>)]]
If I set the input shape when initializing the layer it does not throw any error. So I would guess there is a conflict when using the full input shape (including batch size) and the fit method. Also, the batch size I am setting at the on the layer, is the same as in the fit method. Thank you.

@anayebi

I have 25 videos of 2710 frames(RGB images of size 32x32) each. So there are 3 channels in each frame. How do I build my model to take in video input(all frames of video at once) for the CNN and further the time step for LSTM would be the number of frames. Here is my code that I have written using the keras extras library -

`
model.add(TimeDistributedConvolution2D(32, 3, 3, border_mode='same',input_shape=(2710,2710,img_channels, img_rows, img_cols)))
model.add(keras.layers.normalization.BatchNormalization())
model.add(Activation('relu'))
model.add(TimeDistributedMaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))

model.add(TimeDistributedConvolution2D(32, 3, 3, border_mode='same'))
# model.add(keras.layers.normalization.BatchNormalization())
model.add(Activation('relu'))
model.add(TimeDistributedMaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))

model.add(TimeDistributedConvolution2D(32, 3, 3, border_mode='same'))
# model.add(keras.layers.normalization.BatchNormalization())
model.add(Activation('relu'))
model.add(TimeDistributedMaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))


model.add(TimeDistributedFlatten())
model.add(LSTM(output_dim=255,return_sequences=True))


model.add(TimeDistributedDense(256,activation='relu'))
model.add(TimeDistributedDense(128,activation='relu'))
# model.add(Dense(512,activation='relu'))

model.add(TimeDistributedDense(1,activation='sigmoid'))

sgd = SGD(lr = 0.1, decay = 1e-5, momentum=0.9, nesterov=True)
model.compile(loss='mean_absolute_error',
          optimizer=sgd,
          metrics=['accuracy'])

`

I get this error model.add(TimeDistributedConvolution2D(32, 3, 3, border_mode='same',input_shape=(2710,2710,img_channels, img_rows, img_cols))) File "build/bdist.linux-x86_64/egg/keras/layers/extra.py", line 89, in __init__ AttributeError: can't set attribute

Can you please help me out asap.

Hi,
I am new to Keras and ML . Now I am trying to do some tests about OCR via CNN+LSTM or just LSTM. after searching connect CNN with RNN I found keras-extra . Currently I have hundreds of images and the corresponding words (ground truth). One image can be split into many lines, there should be one word each line at least (demo images as follows). Images after line-segmented may have different height and width. I looked into mnist_cnn.py and mnist_irnn.py, the input is a single letter and images have the same size. the expected output ( label or ground truth) is encoded as numpy array by its index of charset :

digits_letters = u"abcdefghijklmnopqrstuvwxyz"
charset = sorted(list(digits_letters))
"keras" can be encoded as  [10  4 17  0 18] <type 'numpy.ndarray'>

How to set up the model to perform this kind task so that images with different size can be feed into network directly?

Any advice will be welcomed. Thanks in advance!
Best regards,

demo images
page

images after line-segmented
Line-1 labeled "keras"

Line-2 labeled "is a"

Line-3 labeled "great"

Line-4 labeled "tool"