AttentionWrapper States Tuple Out of Bounds (possibly a versioning issue) about keras-utility-layer-collection HOT 6 CLOSED

Hello,
Thanks for reporting this. Can you please add the code which causes the issue? Otherwise I will not be able to solve it, as I have not encountered a similar problem so far.

btw: this has nothing to do with the keras version as I use this package also with version 2.2.0, but most likely with how you construct the layer/model.

from keras-utility-layer-collection.

Hey, thanks for getting back to me so quickly.

Here's some sample code that works for me when I don't use the attention wrapper (and use the encoder final hidden state instead of the hidden sequences).

import keras
import tensorflow as tf
import numpy as np
import kulc.attention


input_1 = keras.layers.Input(shape=(None, 256))

input_2 = keras.layers.Input(shape=(None, ))

encoder = keras.layers.GRU(512, return_state=True, return_sequences=True)

decoder = keras.layers.GRU(512, return_state=True, return_sequences=True)

decoder  =  kulc.attention.AttentionRNNWrapper(decoder)

embedding = keras.layers.Embedding(100, 512)

encoder_outputs, encoder_hidden = encoder(input_1)

input_2_embedding = embedding(input_2)


decoder_output, decoder_hidden = decoder(input_2_embedding,
                                         initial_state=encoder_outputs)

model = keras.Model(inputs=[input_1, input_2], outputs=[decoder_output])


inp_1_example = np.random.rand(2, 100, 256)

inp_2_example = np.array([[1., 2., 3., 4., 5.], [2., 3., 4., 5., 6.]])

ipdb.set_trace()


model.predict(x=[inp_1_example, inp_2_example])

First I get this error:

    "Tensor objects are not iterable when eager execution is not "
TypeError: Tensor objects are not iterable when eager execution is not enabled. To iterate over this tensor use tf.map_fn.

Which I can fix by wrapping encoder_outputs in a list. I then get another interesting error:

import keras
import numpy as np
import kulc.attention


input_1 = keras.layers.Input(shape=(None, 256))

input_2 = keras.layers.Input(shape=(None, ))

encoder = keras.layers.GRU(512, return_state=True, return_sequences=True)

decoder = keras.layers.GRU(512, return_state=True, return_sequences=True)

decoder = kulc.attention.AttentionRNNWrapper(decoder)

embedding = keras.layers.Embedding(100, 512)

encoder_outputs, encoder_hidden = encoder(input_1)

input_2_embedding = embedding(input_2)


decoder_output, decoder_hidden = decoder(input_2_embedding,
                                         initial_state=[encoder_outputs])

model = keras.Model(inputs=[input_1, input_2], outputs=[decoder_output])


inp_1_example = np.random.rand(2, 100, 256)

inp_2_example = np.array([[1., 2., 3., 4., 5.], [2., 3., 4., 5., 6.]])

model.predict(x=[inp_1_example, inp_2_example])

Then I get the error:

  File "test.py", line 25, in <module>
    decoder_output, decoder_hidden = decoder(input_2_embedding,
  File "/root/anaconda3/lib/python3.6/site-packages/keras/engine/base_layer.py", line 460, in __call__
    output = self.call(inputs, **kwargs)
  File "/root/anaconda3/lib/python3.6/site-packages/kulc-0.0.5-py3.6.egg/kulc/attention.py", line 396, in call
    input_length=input_shape[1]
  File "/root/anaconda3/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 2868, in rnn
    outputs, _ = step_function(inputs[0], initial_states + constants)
  File "/root/anaconda3/lib/python3.6/site-packages/kulc-0.0.5-py3.6.egg/kulc/attention.py", line 357, in step
    total_x_prod = states[3]
IndexError: list index out of range

I'm running Keras 2.2.0 and tensorflow-gpu==1.8.0

from keras-utility-layer-collection.

Alright, thanks for the code.

I found a small bug in the kulc code, which is fixed now in package version 0.0.6: I assumed that the RNN cell has two and not only one state. Sadly, I tested the layer only with LSTM cells (which have two states) and not with GRUs (only one state).

Furthermore, I found a mistake in your code, too. You are not passing the encoder_hidden variable but an entire sequence encoder_outputs as the initial state. So please change the way how you call the decoder into this:

decoder_output, decoder_hidden = decoder(input_2_embedding, initial_state=encoder_hidden)

from keras-utility-layer-collection.

Thanks for responding.

The fix appears to work. For what it's worth I had the same problem with an LSTM before but perhaps I made the same mistake re: using the final encoder output state v. using all of the encoder output states.

To be honest I had not read the input feeding paper you cited and did not realize that it was not implementing what Luong et. all are calling "global attention", which is why I used the full set of encoder hidden states.

Thanks for resolving this issue!

from keras-utility-layer-collection.

Ah, I see. Maybe I should expand the documentation slightly in future versions.

You can perform the so-called "global attention" using the ExternalAttentionRNNWrapper layer in kulc.

from keras-utility-layer-collection.

hey zimmerrol
When compiling the model I am getting this error

File "train.py", line 155, in training_model, inference_model, initial_state_inference_model = create_model(le._vocabulary_size, EMBEDDING_SIZE, None, 14*14, 512) File "/home/alok/new-way-video/model.py", line 36, in create_model attented_encoder = ExternalAttentionRNNWrapper(encoder)#, return_attention=True) File "/home/alok/myenv/lib/python3.6/site-packages/kulc/attention.py", line 516, in init super(ExternalAttentionRNNWrapper, self).init(layer, **kwargs) File "/home/alok/myenv/lib/python3.6/site-packages/tensorflow/python/keras/layers/wrappers.py", line 51, in init assert isinstance(layer, Layer) AssertionError

The error is here encoder layer is fine but there is some problem with ExternalAttentionRNNWrapper
encoder = LSTM(embedding_size, return_sequences=True, return_state=False, recurrent_dropout=0.1)
attented_encoder = ExternalAttentionRNNWrapper(encoder, return_attention=True)

hope you will reply soon

from keras-utility-layer-collection.