hy2850 / sequence-generation-pytorch Goto Github PK

Given a first sequence of a sine wave, let the model generate the rest of the wave

Keras LSTM model to learn raw sine waves.

Subtle changes on 'n_steps' and 'hid_dim' alters the fit of the model greatly -> must find a balance between bias (underfitting) and variance (overfitting)

Always start with a simpler problem. Pytorch LSTM model to learn simple number sequence of length 3.

model with single LSTM layer + single FC layer seems to underfit;

bigger hid_dim == more accurate the sequence generation (epoch 1000)

around hid_dim 200 (left), generation accuracy is at tolerable level around 500 (right), maybe the model starts to overfit

-> add one more FC layer to incrase complexity? (LSTM_2FC)

Doesn't seem like working well (flats out or shows huge bias). Maybe another FC layer is an overkill for learning simple sequence data?

1. Using simple LSTM model, train raw sine wave (epoch 200, Grid-layout of hid_dim)

prediction is pretty accurate.

2. Now, train sine wave with Gaussian noise

Training data

less accurate sine wave is generated now (single exp - epoch 200, hid_dim 50)

perhaps, more complex model is able to learn more complete sine wave from the noises? → Not really. Now, hid_dim 20 learns best

• Find appropriate model capacity
  • Changes every time according to the type of data
  • Use grid layout for initial search, then hand tuning
• Train for enough epochs to reduce loss as low as possible
• Using SGD gave flat-out result (underfitting perhaps) -> Changed to Adam to make model converge

optimizer matters!

• Additional hyperparameter tuning can be done in the similar way (learning rate, regularization, ...)