Comments (14)
@nicholas-leonard Thanks for the advice (and super fast response). I'll see if normalization makes a difference here. At least I'm using the modules correctly. I added clipping on the gradients and managed to get the MSE much lower than the baseline. However, I still supposed that the LSTM should be able to do a better job learning addition (adding more hidden units seems like a lot of parameters for a simple function). I'm working on an analogous problem with very long documents (but first layer word embeddings). Are there any tips on initialization for the models? I see many use cases that set weights to uniform parameters.
Thanks again,
Daniel
from rnn.
Hi @shuokay, thank you for your question. So if I understand correctly, you want to input a sequence of images (handwritten characters) into the LSTM to predict a sequence characters (using SoftMax)?
from rnn.
@nicholas-leonard thanks for your reply. the input is a sequence of points that every stroke in one charater throughed, not images, and the output is ONE charater. (the 'online' means you can get every point in one stroke exactly while 'offline' means you can't get the points exactly, what you can get is only one image)
I think it is a many to one task described in this page http://karpathy.github.io/2015/05/21/rnn-effectiveness/
from rnn.
For many to one problems, you can use this kind of architecture :
rnn = nn.Sequential()
rnn:add(nn.Sequencer(nn.Recurrent(...)))
rnn:add(nn.SelectTable(-1))
rnn:add(nn.Linear(...))
rnn:add(nn.LogSoftMax())
So basically, input is a sequence of points (coordinates or whatever), output is log-likelihood of character. The key is the SelectTable(-1)
which only takes the last step's rnn output to predict the character.
from rnn.
@nicholas-leonard thank you very much. Just one advice, Torch and its packages need more detailed documentation.^_^
from rnn.
Hi @nicholas-leonard , still get errors. My code like this:
require 'dp'
require 'torch'
rho=5
hiddenSize=100
inputSize=2
outputSize=10
lstm=nn.LSTM(inputSize, hiddenSize,rho)
model = nn.Sequential()
model:add(lstm)
model:add(nn.SelectTable(-1))
model:add(nn.Linear(hiddenSize,outputSize))
model:add(nn.LogSoftMax())
criterion=nn.ClassNLLCriterion()
input=torch.Tensor({{0,0,1,1,2,2,3,3}}) --the input points in one stroke (0,0),(1,1),(2,2),(3,3)
output=torch.Tensor({1,0,0,0,0,0,0,0,0,0})
model:forward(input[1])
and it give errors:
/home/gys/torch/install/share/lua/5.1/nn/Linear.lua:39: size mismatch, [10 x 2], [8] at /home/gys/torch/pkg/torch/lib/TH/generic/THTensorMath.c:527
stack traceback:
[C]: in function 'addmv'
/home/gys/torch/install/share/lua/5.1/nn/Linear.lua:39: in function 'updateOutput'
/home/gys/torch/install/share/lua/5.1/nn/ParallelTable.lua:12: in function 'updateOutput'
/home/gys/torch/install/share/lua/5.1/nn/Sequential.lua:44: in function 'updateOutput'
/home/gys/torch/install/share/lua/5.1/nn/ConcatTable.lua:11: in function 'updateOutput'
/home/gys/torch/install/share/lua/5.1/nn/Sequential.lua:44: in function 'updateOutput'
/home/gys/torch/install/share/lua/5.1/nn/ConcatTable.lua:11: in function 'updateOutput'
/home/gys/torch/install/share/lua/5.1/nn/Sequential.lua:44: in function 'updateOutput'
/home/gys/torch/install/share/lua/5.1/nn/ConcatTable.lua:11: in function 'updateOutput'
/home/gys/torch/install/share/lua/5.1/nn/Sequential.lua:44: in function 'updateOutput'
/home/gys/torch/install/share/lua/5.1/nnx/LSTM.lua:170: in function 'updateOutput'
/home/gys/torch/install/share/lua/5.1/nn/Sequential.lua:44: in function 'forward'
[string "require 'dp'..."]:15: in main chunk
[C]: in function 'xpcall'
/home/gys/torch/install/share/lua/5.1/itorch/main.lua:177: in function </home/gys/torch/install/share/lua/5.1/itorch/main.lua:143>
/home/gys/torch/install/share/lua/5.1/lzmq/poller.lua:75: in function 'poll'
/home/gys/torch/install/share/lua/5.1/lzmq/impl/loop.lua:307: in function 'poll'
/home/gys/torch/install/share/lua/5.1/lzmq/impl/loop.lua:325: in function 'sleep_ex'
/home/gys/torch/install/share/lua/5.1/lzmq/impl/loop.lua:370: in function 'start'
/home/gys/torch/install/share/lua/5.1/itorch/main.lua:344: in main chunk
[C]: in function 'require'
[string "arg={'/home/gys/.ipython/profile_torch/securi..."]:1: in main chunk
from rnn.
require 'dp'
require 'torch'
hiddenSize=100
inputSize=2
outputSize=10
lstm=nn.LSTM(inputSize, hiddenSize)
model = nn.Sequential()
model:add(nn.SplitTable(1,2))
model:add(nn.Sequencer(lstm))
model:add(nn.SelectTable(-1))
model:add(nn.Linear(hiddenSize,outputSize))
model:add(nn.LogSoftMax())
criterion=nn.ClassNLLCriterion()
input=torch.Tensor({{0,0},{1,1},{2,2},{3,3}}) --the input points in one stroke (0,0),(1,1),(2,2),(3,3)
output=torch.Tensor({1,0,0,0,0,0,0,0,0,0})
model:forward(input)
from rnn.
@nicholas-leonard everything goes well ,thank you very. I will close this issue.
from rnn.
@nicholas-leonard I have a question along similar lines. Do you mind telling me what is wrong with my training here? In this toy example, the RNN doesn't perform well learning the sum of a series.
-- more imports than necessary
require 'nn'
require 'cunn'
require 'rnn'
require 'dp'
require 'cutorch'
-- set to a small learning rate due to possible explosion
lr = .00001
-- roll lstm over the numbers in the sequence, select last output layer, apply linear model
rnn = nn.Sequential()
lstm = nn.FastLSTM( 1,500)
rnn:add( nn.Sequencer( lstm) )
rnn:add( nn.SelectTable(-1) )
rnn:add( nn.Linear(500,1) )
rnn:cuda()
criterion = nn.MSECriterion():cuda()
-- random numbers that are scaled to make the problem a little harder
inputs = torch.rand(100,200)
for i=1,100 do
inputs[i] = inputs[i]* i
end
inputs = inputs:cuda()
targets = inputs:sum(2):cuda()
baseline = targets:std()^2
print( baseline ) -- usually around 8,600,000
for i =1,10000 do
rnn:training()
errors = {}
for j = 1,100 do
--get input row and target
local input = inputs[j]:split(1)
local target = targets[j]
-- print( target)
local output = rnn:forward( input)
local err = criterion:forward( output, target)
table.insert( errors, err)
local gradOutputs = criterion:backward(output, target)
rnn:backward(input, gradOutputs)
rnn:updateParameters(lr)
rnn:zeroGradParameters()
rnn:forget()
end
print ( torch.mean( torch.Tensor( errors)) )
end
from rnn.
@flybass Your problem is very hard. Maybe the model needs more capacity. You could try adding more hidden (1000 instead of 500), and/or stacking another LSTM over the first one.
Also, you could try it with more examples. 100 is not a lot. Also, the data is unbounded. So a sum could be 1, or it could be 10000. If you could find a way to keep it within a range, say between -1 and 1, that might help. To do so, you could normalize the dataset so that the min and max sum is -1 and 1 respectively.
from rnn.
@flybass You can try the default initialization, or getParameters():uniform(0.1) (or something around that number). It's empirical, so trial and error. But usually, I find uniform 0.1 to work for me.
from rnn.
@nicholas-leonard
Hey Nick,
I spoke to a professor about this issue and he mentioned to initialize the LSTM weights so they are orthogonal. I'll compare this method.
from rnn.
@flybass let me know if that works best. Also, if it does work, you could add a initOrthogonal()
to LSTM !
from rnn.
Is there a way to initialize the rnn for a one to many sequence? I am doing a single output to six output system Everything is fine until I call gradInput = rnn:backward(inputs[step], gradOutputs[step])
. It keeps complaining that inputs
size should be equal in size to gradOutputs
. I do not expect them to be equal. I noticed this is due to an assert function that was built into the Linear
and nn
modules. Would you be able to look into the code below for me please? I have
require 'torch'
require 'nn'
ninputs = 1
noutputs = 6
nhiddens_rnn = 1
start = 1 -- the size of the output (excluding the batch dimension)
rnnInput = nn.Linear(ninputs, start) --the size of the output
feedback = nn.Linear(start, ninputs) --module that feeds back prev/output to transfer module
transfer = nn.ReLU()
--RNN
r = nn.Recurrent(start,
rnnInput, --input module from inputs to outs
feedback,
transfer,
rho
)
neunet = nn.Sequential()
:add(r)
:add(nn.Linear(nhiddens_rnn, noutputs))
neunet = nn.Sequencer(neunet)
print('rnn')
print(neunet)
I train as follows in mini-batches
local offsets = {}
--form mini batch
for t = 1, train_input:size()[1], 1 do
for i = t, t+opt.batchSize-1 do
table.insert(offsets, train_input[i])
end
offsets = torch.cat({offsets[1], offsets[2], offsets[3], offsets[4],
offsets[5], offsets[6]})
-- print('offsets b4'); print(offsets)
offsets = torch.LongTensor():resize(offsets:size()[1]):copy(offsets)
--BPTT
local iter = 1
-- 1. create a sequence of rho time-steps
local inputs, targets = {}, {}
for step = 1, rho do
--batch of inputs
inputs[step] = train_input:index(1, offsets)
--batch of targets
targets[step] = {train_out[1]:index(1, offsets), train_out[2]:index(1, offsets),
train_out[3]:index(1, offsets), train_out[4]:index(1, offsets),
train_out[5]:index(1, offsets), train_out[6]:index(1, offsets)}
--increase offsets indices by 1
offsets = train_input[{ {t+step, t+step+rho} }]
offsets = torch.LongTensor():resize(offsets:size()[1]):copy(offsets)
end
--2. Forward sequence through rnn
neunet:zeroGradParameters()
neunet:forget() --forget all past time steps
local outputs, err = {}, 0
local inputs_ = {}
local targetsTable = {}
for step = 1, rho do
table.insert(inputs_, inputs[step])
outputs[step] = neunet:forward(inputs_)
_, outputs[step] = catOut(outputs, step, noutputs, opt)
--reshape output data
_, targetsTable = catOut(targets, step, noutputs, opt)
err = err + cost:forward(outputs[step], targetsTable)
print('err', err)
end
print(string.format("Step %d, Loss error = %f ", iter, err ))
--3. do backward propagation through time(Werbos, 1990, Rummelhart, 1986)
local gradOutputs, gradInputs = {}, {}
local inputs_bkwd = {}
for step = rho, 1, -1 do --we basically reverse order of forward calls
gradOutputs[step] = cost:backward(outputs[step], targets[step])
--resize inputs before backward call
inputs_bkwd = gradInputResize(inputs, step, noutputs, opt)
gradInputs[step] = neunet:backward(inputs_bkwd, gradOutputs[step])
-- print('gradInputs'); print(gradInputs)
end
--4. update lr
neunet:updateParameters(opt.rnnlearningRate)
iter = iter + 1
end
I am getting errors when I call neunet:backward(inputs_bkwd, gradOutputs[step])
:
home/local/ANT/ogunmolu/torch/install/bin/luajit: ...l/ANT/ogunmolu/torch/install/share/lua/5.1/nn/Linear.lua:75: size mismatch, m1: [6 x 1], m2: [6 x 1] at /home/local/ANT/ogunmolu/torch/pkg/torch/lib/TH/generic/THTensorMath.c:706
stack traceback:
[C]: in function 'addmm'
...l/ANT/ogunmolu/torch/install/share/lua/5.1/nn/Linear.lua:75: in function 'updateGradInput'
...T/ogunmolu/torch/install/share/lua/5.1/nn/Sequential.lua:55: in function 'updateGradInput'
...NT/ogunmolu/torch/install/share/lua/5.1/rnn/Recursor.lua:45: in function '_updateGradInput'
...lu/torch/install/share/lua/5.1/rnn/AbstractRecurrent.lua:46: in function 'updateGradInput'
...T/ogunmolu/torch/install/share/lua/5.1/rnn/Sequencer.lua:78: in function 'updateGradInput'
...l/ANT/ogunmolu/torch/install/share/lua/5.1/nn/Module.lua:30: in function 'backward'
rnn.lua:475: in function 'train'
rnn.lua:688: in main chunk
[C]: in function 'dofile'
...molu/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:145: in main chunk
[C]: at 0x00406240
Would appreciate your help!
I left a gist here in case you are interested in the code.
from rnn.
Related Issues (20)
- Weight normalization
- SequencerCriterion does not appear to work in batch mode HOT 1
- Error when using dropout in GRU HOT 1
- Problem at inference with Sequencer HOT 1
- Make rnn to pytorch? HOT 1
- Bad size of gradInput in BiSequencerLM HOT 1
- invalid parent class name nn.Decorator HOT 6
- rnn/Sequencer.lua:90: attempt to index field 'tableoutput' (a nil value) HOT 2
- [Feature request] nn.seqBRNN ability to stack and maskZero()
- attempt to call field 'Recurrent' (a nil value) HOT 1
- Error: class nn.SpatialGlimpse has been already assigned a parent class HOT 2
- Evaluation error in "evaluate-rva.lua". HOT 1
- Why "gradOutput" buffers are not freed after calling clearState()?
- Character-level BiLSTM take the first and last hidden state HOT 2
- rnn init.lua contains classes that belong to dpnn HOT 2
- error: not enough memory when running noise-contrastive-estimate example
- Error when recurrent visual attention model on sample data non MNIST
- ParallelTable with Sequencer does not seem to work HOT 2
- Module to add soft/hard attention mechanism to a LSTM
- Doubt regarding MaskZero
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from rnn.