LSTM_Captioning using minpy: memory issue about minpy HOT 3 CLOSED

The overflow is due to the gradient definition of tanh (g / np.cosh(x) ** 2). It seems that the divisor is too small which leads to numerical overflow. Let me see whether I could implement it in a more stable way. Thanks for the reporting!

For memory issue, @hotpxl could you have a look. I remember there is some weak reference problem in the autograd part before.

from minpy.

Similar issue detected in the rnn perf test. Please check the code: https://github.com/dmlc/minpy/blob/rnn_perf/examples/nn/rnn_test/rnn_minpy_perf.py (you should copy the file to examples/nn folder to run)

I did some math on how much memory should we use:
Input： 256, Hidden: 2560, Out: 1, seq_len: 30, batch_size: 100

Weights:
Wx = 256 * 2560 * 4 / 1024 / 1024 = 2.5M
b = 2560 * 4 / 1024 / 1024 = 0.01M
Wh = 2560 * 2560 * 4 / 1024 / 1024 = 25M
hb = 0.01M
Wout = 0.01M
Sum = 27.03M

Activation:
Input: 100 * 256 * 30 * 4 / 1024 / 1024 = 0.3M
Activation: 100 * 2560 * 30 * 4 / 1024 / 1024 = 29.3M
Sum: 29.6M

When involving BP, just double the space for error derivative and gradients for the weigts, and if we have momentum, just add another pie of weights. Total memory needed should be about:
29.6 * 2 + 27.03 * 3 = 59.2 + 81.08 = 140M

The minpy example seems run more than 1500M on my device.

from minpy.

This should been solved in #112 #117 .

from minpy.