The nilm's discuss from goruck

nilm's Issues

PositionwiseFeedForward operation order

Hello, thank you for providing this repository.

I was wondering if in the following code the activation function should be applied to x_fc1?
return self.fully_connected2(self.activation(x_fc1)) instead of return self.activation(self.fully_connected2(x_fc1))

original code lin 326:
https://github.com/goruck/nilm/blob/main/ml/transformer_model.py

class PositionwiseFeedForward(tf.keras.layers.Layer):
    """PositionwiseFeedForward layer.

    This is an implementation of position wise feed-forward as described in the
    paper "Attention is all you Need" (Vaswani et al., 2017)."""

    def __init__(self, d_model, d_ff, **kwargs):
        super().__init__(**kwargs)
        self.fully_connected1 = tf.keras.layers.Dense(d_ff)  # First fully connected layer
        self.fully_connected2 = tf.keras.layers.Dense(d_model)  # Second fully connected layer
        self.activation = GELU()

    def call(self, inputs, *args, **kwargs):
        # The input is passed into the two fully-connected layers, with GELU activation.
        x_fc1 = self.fully_connected1(inputs)

        return self.activation(self.fully_connected2(x_fc1))

Recommend Projects