Shakespeare - no decent results. about grenade HOT 1 OPEN

Hi Celina,

This one unfortunately takes a long time.

There are a few reasons for this, the biggest of which is minibatching. For convolution NN (which was the first proving ground for grenade), almost all non-trivial computations are matrix matrix multiplications, so minibatching doesn't give a huge computational benefit, and I left it out for semantic clarity; for LSTMs however, it's mostly matrix vector multiplications, which become matrix matrix with minibatching. This makes a big difference actually, as a 50 column wide matrix takes only about 5 times as much as a matrix vector op with decent BLAS libraries.

The other two things I have done differently are not propagating the previous batch's input vector (to line them up for truncated back-prop through time) and using only SGD with momentum instead of ADAM or some other optimiser.

If memory serves, I trained for over 24 hours, slowly ramping up the training size, starting with about 15 characters until spaces were interspersed well, then working up to 25 and 50 when words and basic grammar appeared.

Huw

from grenade.