hanxiao / daanet Goto Github PK
View Code? Open in Web Editor NEWDAANet: Dual Ask-Answer Network for Machine Reading Comprehension
Home Page: https://arxiv.org/abs/1809.01997
DAANet: Dual Ask-Answer Network for Machine Reading Comprehension
Home Page: https://arxiv.org/abs/1809.01997
try:
shutil.copytree('./', hparams.get('code_dir'), ignore=shutil.ignore_patterns(*IGNORE_PATTERNS))
logger.info('current code base is copied to %s' % hparams.get('save_dir'))
except FileExistsError:
logger.info('code base exist, no need to copy!')
Maybe something wrong with these?
Windows 7.
Can you please elaborate more for steps to train, test and evaluate. And things like where the data is stored, approximately how much time it takes for metrics evaluation. It would be really of a good help to everyone.
...
I:0:1027-182128:[hel:par:166]: task_name = qag
I:0:1027-182128:[hel:par:166]: test_files = ['./sample_data/sample.json']
I:0:1027-182128:[hel:par:166]: train_files = ['./sample_data/sample.json']
I:0:1027-182128:[hel:par:166]: use_answer_masks = True
I:0:1027-182128:[hel:par:166]: use_coverage = True
I:0:1027-182128:[hel:par:166]: use_oovs = True
I:0:1027-182128:[hel:par:166]: weight_decay = 0
I:0:1027-182128:[hel:par:166]:word_embedding_files = ['./sample_data/sample.embed.txt']
I:0:1027-182128:[voc:loa: 87]:loading word embedding from ./sample_data/sample.embed.txt
I:0:1027-182128:[voc:loa:108]:building token-id map...
I:0:1027-182128:[voc:loa:130]:size of embedding 1004 x 256
I:0:1027-182128:[voc:loa:132]:size of pretrain embedding 1000 x 256
Traceback (most recent call last):
File "app.py", line 15, in <module>
run()
File "app.py", line 11, in run
getattr(__import__('api'), sys.argv[1])(args)
File "/home/alek/Desktop/daanet-env/daanet-2/api.py", line 15, in train
model = build_model(args)
File "/home/alek/Desktop/daanet-env/daanet-2/utils/helper.py", line 323, in build_model
return rccore.RCCore(args)
File "/home/alek/Desktop/daanet-env/daanet-2/daanet/basic.py", line 24, in __init__
super().__init__(args)
File "/home/alek/Desktop/daanet-env/daanet-2/daanet/base.py", line 16, in __init__
super().__init__(args)
File "/home/alek/Desktop/daanet-env/daanet-2/base/base_model.py", line 30, in __init__
self.data_io = dataio.DataIO(args)
File "/home/alek/Desktop/daanet-env/daanet-2/dataio_utils/full_load_io.py", line 14, in __init__
ModeKeys.TRAIN: self.load_data(self.args.train_files, ModeKeys.TRAIN),
File "/home/alek/Desktop/daanet-env/daanet-2/base/base_io.py", line 23, in load_data
raise NotImplementedError
NotImplementedError
Originally posted by @alekzieba in #2 (comment)
Unfortunately training doesn't work for me, I'm getting the following error (abbreviated):
use config: default_gpu I:[helper.py:188]:number of jobs in the queue: 1 I:[helper.py:190]:will start the job: python app.py train /tmp/daanet-sb_vvrih ... use config: default_gpu I:0:0915-124247:[hel:par:119]:loading parameters... I:0:0915-124247:[hel:par:128]:[add from /tmp/daanet-hfxaocgi] task_name: qag I:0:0915-124247:[hel:par:128]:[add from /tmp/daanet-hfxaocgi] train_files: ['./sample_data/sample.json'] I:0:0915-124247:[hel:par:128]:[add from /tmp/daanet-hfxaocgi] dev_files: ['./sample_data/sample.json'] I:0:0915-124247:[hel:par:128]:[add from /tmp/daanet-hfxaocgi] test_files: ['./sample_data/sample.json'] I:0:0915-124247:[hel:par:128]:[add from /tmp/daanet-hfxaocgi] word_embedding_files: ['./sample_data/sample.embed.txt'] I:0:0915-124247:[hel:par:128]:[add from /tmp/daanet-hfxaocgi] char_embedding_files: ['./sample_data/sample.charembed.txt'] I:0:0915-124247:[hel:par:128]:[add from /tmp/daanet-hfxaocgi] result_dir: ./sample_data/result Traceback (most recent call last): File "app.py", line 15, in <module> run() File "app.py", line 10, in run args = parse_args(sys.argv[2] if len(sys.argv) > 2 else None, MODEL_ID, CONFIG_SET, followup_args) File "/home/undef/readingComprehension/daanet/utils/helper.py", line 148, in parse_args shutil.copytree('./', hparams.get('code_dir'), ignore=shutil.ignore_patterns(*IGNORE_PATTERNS)) File "/home/undef/readingComprehension/daanet/env/lib/python3.6/shutil.py", line 359, in copytree raise Error(errors) shutil.Error: [('./save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020...
I am trying to run on GPU. But it is showing a message, All GPUs are busy. Waiting for a free slot
. So is there any way to assign GPU to this process and if not how do I run it on CPU?
I'm interested in dual ask-answer model too much.
I weired error in below
File "/Users/sunu/IdeaProjects/QAZeroToAll/daanet/daanet/basic.py", line 169, in _embed
trainable=True)
File "/Users/sunu/IdeaProjects/QAZeroToAll/daanet/nlp/nn.py", line 91, in get_var
regularizer=regularizer_fn, **kwargs)
...
ValueError: You can only pass an initializer function that expects no arguments to its callable when the shape is not fully defined. The given initializer function expects the following args ['shape', 'dtype', 'partition_info']
I wondered why
daanet / basic.py
only use the get_var () method in self.char_emb
There seems to be missing file in dataio_utils
The full_load_io doesn't implement the JSON loader necessary for io of sample data and squad's JSON format.
tokenid2charsid is also not implemented.
AtributeError: 'DataIO' object has no attribute 'tokenid2charsid'.
Can you include the code for a progress bar? This will help to know how much training is completed.
I'm trying to run
python grid_search.py daanet
on a ubuntu 18.04 machine with an anaconda tensorflow environment
ruamel.yaml 0.15.79
when i run "python grid_search.py daanet", i get
`Traceback (most recent call last):
File "grid_search.py", line 43, in
run()
File "grid_search.py", line 20, in run
settings_default = YAML().load(fp)
File "/home/feral/anaconda3/envs/tensorflow/lib/python3.6/site-packages/ruamel/yaml/main.py", line 331, in load
return constructor.get_single_data()
File "/home/feral/anaconda3/envs/tensorflow/lib/python3.6/site-packages/ruamel/yaml/constructor.py", line 108, in get_single_data
return self.construct_document(node)
File "/home/feral/anaconda3/envs/tensorflow/lib/python3.6/site-packages/ruamel/yaml/constructor.py", line 118, in construct_document
for _dummy in generator:
File "/home/feral/anaconda3/envs/tensorflow/lib/python3.6/site-packages/ruamel/yaml/constructor.py", line 1508, in construct_yaml_map
self.construct_mapping(node, data, deep=True)
File "/home/feral/anaconda3/envs/tensorflow/lib/python3.6/site-packages/ruamel/yaml/constructor.py", line 1413, in construct_mapping
value = self.construct_object(value_node, deep=deep)
File "/home/feral/anaconda3/envs/tensorflow/lib/python3.6/site-packages/ruamel/yaml/constructor.py", line 171, in construct_object
for _dummy in generator:
File "/home/feral/anaconda3/envs/tensorflow/lib/python3.6/site-packages/ruamel/yaml/constructor.py", line 1508, in construct_yaml_map
self.construct_mapping(node, data, deep=True)
File "/home/feral/anaconda3/envs/tensorflow/lib/python3.6/site-packages/ruamel/yaml/constructor.py", line 1364, in construct_mapping
merge_map = self.flatten_mapping(node)
File "/home/feral/anaconda3/envs/tensorflow/lib/python3.6/site-packages/ruamel/yaml/constructor.py", line 1315, in flatten_mapping
raise DuplicateKeyError(*args)
ruamel.yaml.constructor.DuplicateKeyError: while constructing a mapping
in "default.yaml", line 66, column 14
found duplicate key "<<"
in "default.yaml", line 68, column 3
To suppress this check see:
http://yaml.readthedocs.io/en/latest/api.html#duplicate-keys
Duplicate keys will become an error in future releases, and are errors
by default when using the new API.
`
Thanks in advance : )
hi, how to run a code on cpu help me..
I was trying to run the code there seem to be multiple errors in utils.helper with HParams (TypeError: argument of type 'HParams' is not iterable
) and the save
folder gets created recursively with the entire code in it all over again multiple times. Could you please update the code with the necessary changes?
Thanks!
Can you please add your requirement.txt please so that it'll be easy to build # #
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.