Implementation of Dynamic memory networks by Kumar et al. http://arxiv.org/abs/1506.07285

License: MIT License

Python 99.59% Shell 0.41%

babi-tasks theano deep-learning natural-language-processing

dynamic-memory-networks-in-theano's Introduction

Dynamic memory networks in Theano

The aim of this repository is to implement Dynamic memory networks as described in the paper by Kumar et al. and to experiment with its various extensions.

Pretrained models on bAbI tasks can be tested online.

We will cover the process in a series of blog posts.

The first post describes the details of the basic architecture and presents our first results on bAbI tasks v1.2.
The second post describes our second model called dmn_smooth and introduces our playground for bAbI tasks.

Repository contents

file	description
`main.py`	the main entry point to train and test available network architectures on bAbI-like tasks
`dmn_basic.py`	our baseline implementation. It is as close to the original as we could understand the paper, except the number of steps in the main memory GRU is fixed. Attention module uses `T.abs_` function as a distance between two vectors which causes gradients to become `NaN` randomly. The results reported in this blog post are based on this network
`dmn_smooth.py`	uses the square of the Euclidean distance instead of `abs` in the attention module. Training is very stable. Performance on bAbI is slightly better
`dmn_batch.py`	`dmn_smooth` with minibatch training support. The batch size cannot be set to `1` because of the Theano bug
`dmn_qa_draft.py`	draft version of a DMN designed for answering multiple choice questions
`utils.py`	tools for working with bAbI tasks and GloVe vectors
`nn_utils.py`	helper functions on top of Theano and Lasagne
`fetch_babi_data.sh`	shell script to fetch bAbI tasks (adapted from MemN2N)
`fetch_glove_data.sh`	shell script to fetch GloVe vectors (by 5vision)
`server/`	contains Flask-based restful api server

Usage

This implementation is based on Theano and Lasagne. One way to install them is:

pip install -r https://raw.githubusercontent.com/Lasagne/Lasagne/master/requirements.txt
pip install https://github.com/Lasagne/Lasagne/archive/master.zip

The following bash scripts will download bAbI tasks and GloVe vectors.

./fetch_babi_data.sh
./fetch_glove_data.sh

Use main.py to train a network:

python main.py --network dmn_basic --babi_id 1

The states of the network will be saved in states/ folder. There is one pretrained state on the 1st bAbI task. It should give 100% accuracy on the test set:

python main.py --network dmn_basic --mode test --babi_id 1 --load_state states/dmn_basic.mh5.n40.babi1.epoch4.test0.00033.state

Server

If you want to start a server which will return the predication for bAbi tasks, you should do the following:

Generate UI files as described in YerevaNN/dmn-ui
Copy the UI files to server/ui
Run the server

cd server && python api.py

If have Docker installed, you can pull our Docker image with ready DMN server.

docker pull yerevann/docker
docker run --name dmn_1 -it --rm -p 5000:5000 yerevann/dmn

Roadmap

Mini-batch training (done, 08/02/2016)
Web interface (done, 08/23/2016)
Visualization of episodic memory module (done, 08/23/2016)
Regularization (work in progress, L2 doesn't help at all, dropout and batch normalization help a little)
Support for multiple-choice questions (work in progress)
Evaluation on more complex datasets
Import some ideas from Neural Reasoner

License

dynamic-memory-networks-in-theano's People

Contributors

Stargazers

Watchers

Forkers

kastnerkyle fsodogandji vazgenh anirudh9119 rtvt123 wgapl jeffzhengye amoliu ml-lab g-wang poyuwu asmith26 beronx86 zh4ngx sigmaquan chanhou shailja11 chagge dapeng2018 daedalusx cheripai yorkerlin liuxuvip cyloss sophie-germain eunchung xiuxiulu qinggege arnabgho kirk86 lijian8 bibutahseen codeaudit willemjan homerquan angelo337 fxfactorial kevinwenya duankai nicholasyuan qyouurcs vyraun vijaysudheer mjc14 tianlongwang beige90 armendarbinyan carsonkahn-external reflen oneproton bexcite highsmallxu kratarth1203 zshwuhan moses1994 everglowing hongyunnchen hansonboy adrianhust nishnik chun5212021202 hfxunlp makai281 sszzsupersupersupersuper tonytongzhao shuaidong-jiao bauerzhou botma anthonylife wangjin0818 ponlee jtkostman falconzyx lanshuofeng raghavendranpm styxjedi karthy1988 geeky-bit ppjerry pzhao16me shubhampachori12110095 rosssong ms8909 binbinbian renyi533 coldzoo petrosalema ntson2002 noble6emc2 janeshenyy malcolmgreaves imdahmash zabihimayvan qingwang12138 mahimajeslani26 satadru5 gluckez vaishnavi-amalladinne fangzheng354 afcarl

dynamic-memory-networks-in-theano's Issues

Memory usage of this model

Hi, I try to run this model on a 500M+ corpus, It breaks after all my 16G memory and 16G swap run out while compiling train_fn. Is there any way to run this model on my corpus?

where is the dmn_batch code ?

where is the dmn_batch code ? i couldn't find it the project

New Facts and questions

If we add new facts, does the model need to be retrained? I am getting inaccurate answers for non Sample Questions

Story:
Sandra moved to the garden.
John went to the bedroom.
John moved to the bathroom.
John went to the garden.
John journeyed to the bedroom.
Sandra went to the office.
Sandra went to the bathroom.
Sandra moved to the kitchen.
Apple is red in color

Question
What color is the apple?

It gives a wrong answer?
Any thoughts?

Dynamic-memory-networks-in-Theano/data/en/all_shuffled_train.txt not found

==> Loading test from /private/tmp/new/Dynamic-memory-networks-in-Theano/data/en/all_shuffled_train.txt
Traceback (most recent call last):
  File "api.py", line 73, in <module>
    babi_train_raw, babi_test_raw = dmn_utils.get_babi_raw('joint', 'joint')
  File "../utils.py", line 78, in get_babi_raw
    babi_train_raw = init_babi(os.path.join(os.path.dirname(os.path.realpath(__file__)), 'data/en/%s_train.txt' % babi_name))
  File "../utils.py", line 8, in init_babi
    for i, line in enumerate(open(fname)):
IOError: [Errno 2] No such file or directory: '/private/tmp/new/Dynamic-memory-networks-in-Theano/data/en/all_shuffled_train.txt'

when running

python api.py

server/

The link to dmn-ui is broken.

"Generate UI files as described in YerevaNN/dmn-ui"

The above link is broken. Can you please help to point to the correct url. Thanks

TypeError: float() argument must be a string or a number, not 'map'

While testing the model using command on cmd:

python main.py --network dmn_basic --mode test --babi_id 1 --load_state states/dmn_basic.mh5.n40.babi1.epoch4.test0.00033.state

the error which occurs is:

Traceback (most recent call last):
  File "main.py", line 72, in <module>
    dmn = dmn_basic.DMN_basic(**args_dict)
  File "C:\Users\Karan\Desktop\DMT\dmn_basic.py", line 38, in __init__
    self.train_input, self.train_q, self.train_answer, self.train_input_mask = self._process_input(babi_train_raw)
  File "C:\Users\Karan\Desktop\DMT\dmn_basic.py", line 299, in _process_input
    inputs.append(np.vstack(inp_vector).astype(floatX))
TypeError: float() argument must be a string or a number, not 'map'

Whole CMD output is:

C:\Users\Karan\Desktop\DMT>python main.py --network dmn_basic --mode test --babi_id 1 --load_state states/dmn_basic.mh5.n40.babi1.epoch4.test0.00033.state
C:\Users\Karan\AppData\Local\Programs\Python\Python36\lib\site-packages\numpy\core\__init__.py:29: UserWarning: loaded more than 1 DLL from .libs:
C:\Users\Karan\AppData\Local\Programs\Python\Python36\lib\site-packages\numpy\.libs\libopenblas.IPBC74C7KURV7CB2PKT5Z5FNR3SIBV4J.gfortran-win_amd64.dll
C:\Users\Karan\AppData\Local\Programs\Python\Python36\lib\site-packages\numpy\.libs\libopenblas.TXA6YQSD3GCQQC22GEQ54J2UDCXDXHWN.gfortran-win_amd64.dll
  stacklevel=1)
WARNING (theano.configdefaults): g++ not available, if using conda: `conda install m2w64-toolchain`
C:\Users\Karan\AppData\Local\Programs\Python\Python36\lib\site-packages\theano\configdefaults.py:560: UserWarning: DeprecationWarning: there is no c++ compiler.This is deprecated and with Theano 0.11 a c++ compiler will be mandatory
  warnings.warn("DeprecationWarning: there is no c++ compiler."
WARNING (theano.configdefaults): g++ not detected ! Theano will be unable to execute optimized C-implementations (for both CPU and GPU) and will default to Python implementations. Performance will be severely degraded. To remove this warning, set Theano flags cxx to an empty string.
WARNING (theano.tensor.blas): Using NumPy C-API based implementation for BLAS functions.
==> parsing input arguments
Namespace(answer_module='feedforward', babi_id='1', babi_test_id='', batch_norm=False, batch_size=10, dim=40, dropout=0.0, epochs=500, input_mask_mode='sentence', l2=0, load_state='states/dmn_basic.mh5.n40.babi1.epoch4.test0.00033.state', log_every=1, memory_hops=5, mode='test', network='dmn_basic', normalize_attention=False, prefix='', save_every=1, shuffle=True, word_vector_size=50)
==> Loading test from C:\Users\Karan\Desktop\DMT\data/en/qa1_single-supporting-fact_train.txt
==> Loading test from C:\Users\Karan\Desktop\DMT\data/en/qa1_single-supporting-fact_test.txt
==> loading glove
==> glove is loaded
==> no minibatch training, argument batch_size is useless
==> not used params in DMN class: dict_keys(['babi_id', 'babi_test_id', 'batch_norm', 'batch_size', 'dropout', 'epochs', 'load_state', 'log_every', 'network', 'prefix', 'save_every', 'shuffle'])
Traceback (most recent call last):
  File "main.py", line 72, in <module>
    dmn = dmn_basic.DMN_basic(**args_dict)
  File "C:\Users\Karan\Desktop\DMT\dmn_basic.py", line 38, in __init__
    self.train_input, self.train_q, self.train_answer, self.train_input_mask = self._process_input(babi_train_raw)
  File "C:\Users\Karan\Desktop\DMT\dmn_basic.py", line 299, in _process_input
    inputs.append(np.vstack(inp_vector).astype(floatX))
TypeError: float() argument must be a string or a number, not 'map'

Error is in this section of code in dwn_basic.py and at _process_input function

inputs.append(np.vstack(inp_vector).astype(floatX)) questions.append(np.vstack(q_vector).astype(floatX))

i have solved all the print statement related errors, i am using _pickle instead of cPickle as cPickle was in python 2 and i am currently on python 3.
another error which i solved is:

UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 2273: character maps to <undefined>

by adding encoding="utf8" in utils.py under load_glove(dim) function
This is the changed code:
with open(os.path.join(os.path.dirname(os.path.realpath(__file__)), "data/glove/glove.6B." + str(dim) + "d.txt"),encoding="utf8") as f:

I want to know how to solve (not 'map') above error and are my changes done to code responsible for it?

yerevann / dynamic-memory-networks-in-theano Goto Github PK