hanxiao / daanet Goto Github PK

try:
    shutil.copytree('./', hparams.get('code_dir'), ignore=shutil.ignore_patterns(*IGNORE_PATTERNS))
    logger.info('current code base is copied to %s' % hparams.get('save_dir'))
except FileExistsError:
    logger.info('code base exist, no need to copy!')

Maybe something wrong with these?
Windows 7.

...
I:0:1027-182128:[hel:par:166]:           task_name = qag                 
I:0:1027-182128:[hel:par:166]:          test_files = ['./sample_data/sample.json']
I:0:1027-182128:[hel:par:166]:         train_files = ['./sample_data/sample.json']
I:0:1027-182128:[hel:par:166]:    use_answer_masks = True                
I:0:1027-182128:[hel:par:166]:        use_coverage = True                
I:0:1027-182128:[hel:par:166]:            use_oovs = True                
I:0:1027-182128:[hel:par:166]:        weight_decay = 0                   
I:0:1027-182128:[hel:par:166]:word_embedding_files = ['./sample_data/sample.embed.txt']
I:0:1027-182128:[voc:loa: 87]:loading word embedding from ./sample_data/sample.embed.txt
I:0:1027-182128:[voc:loa:108]:building token-id map...
I:0:1027-182128:[voc:loa:130]:size of embedding 1004 x 256
I:0:1027-182128:[voc:loa:132]:size of pretrain embedding 1000 x 256
Traceback (most recent call last):
  File "app.py", line 15, in <module>
    run()
  File "app.py", line 11, in run
    getattr(__import__('api'), sys.argv[1])(args)
  File "/home/alek/Desktop/daanet-env/daanet-2/api.py", line 15, in train
    model = build_model(args)
  File "/home/alek/Desktop/daanet-env/daanet-2/utils/helper.py", line 323, in build_model
    return rccore.RCCore(args)
  File "/home/alek/Desktop/daanet-env/daanet-2/daanet/basic.py", line 24, in __init__
    super().__init__(args)
  File "/home/alek/Desktop/daanet-env/daanet-2/daanet/base.py", line 16, in __init__
    super().__init__(args)
  File "/home/alek/Desktop/daanet-env/daanet-2/base/base_model.py", line 30, in __init__
    self.data_io = dataio.DataIO(args)
  File "/home/alek/Desktop/daanet-env/daanet-2/dataio_utils/full_load_io.py", line 14, in __init__
    ModeKeys.TRAIN: self.load_data(self.args.train_files, ModeKeys.TRAIN),
  File "/home/alek/Desktop/daanet-env/daanet-2/base/base_io.py", line 23, in load_data
    raise NotImplementedError
NotImplementedError

Originally posted by @alekzieba in #2 (comment)

I am trying to run on GPU. But it is showing a message, All GPUs are busy. Waiting for a free slot. So is there any way to assign GPU to this process and if not how do I run it on CPU?

Can you please add your requirement.txt please so that it'll be easy to build # #

I'm trying to run
python grid_search.py daanet
on a ubuntu 18.04 machine with an anaconda tensorflow environment
ruamel.yaml 0.15.79

when i run "python grid_search.py daanet", i get
`Traceback (most recent call last):
File "grid_search.py", line 43, in
run()
File "grid_search.py", line 20, in run
settings_default = YAML().load(fp)
File "/home/feral/anaconda3/envs/tensorflow/lib/python3.6/site-packages/ruamel/yaml/main.py", line 331, in load
return constructor.get_single_data()
File "/home/feral/anaconda3/envs/tensorflow/lib/python3.6/site-packages/ruamel/yaml/constructor.py", line 108, in get_single_data
return self.construct_document(node)
File "/home/feral/anaconda3/envs/tensorflow/lib/python3.6/site-packages/ruamel/yaml/constructor.py", line 118, in construct_document
for _dummy in generator:
File "/home/feral/anaconda3/envs/tensorflow/lib/python3.6/site-packages/ruamel/yaml/constructor.py", line 1508, in construct_yaml_map
self.construct_mapping(node, data, deep=True)
File "/home/feral/anaconda3/envs/tensorflow/lib/python3.6/site-packages/ruamel/yaml/constructor.py", line 1413, in construct_mapping
value = self.construct_object(value_node, deep=deep)
File "/home/feral/anaconda3/envs/tensorflow/lib/python3.6/site-packages/ruamel/yaml/constructor.py", line 171, in construct_object
for _dummy in generator:
File "/home/feral/anaconda3/envs/tensorflow/lib/python3.6/site-packages/ruamel/yaml/constructor.py", line 1508, in construct_yaml_map
self.construct_mapping(node, data, deep=True)
File "/home/feral/anaconda3/envs/tensorflow/lib/python3.6/site-packages/ruamel/yaml/constructor.py", line 1364, in construct_mapping
merge_map = self.flatten_mapping(node)
File "/home/feral/anaconda3/envs/tensorflow/lib/python3.6/site-packages/ruamel/yaml/constructor.py", line 1315, in flatten_mapping
raise DuplicateKeyError(*args)
ruamel.yaml.constructor.DuplicateKeyError: while constructing a mapping
in "default.yaml", line 66, column 14
found duplicate key "<<"
in "default.yaml", line 68, column 3

To suppress this check see:
http://yaml.readthedocs.io/en/latest/api.html#duplicate-keys

Duplicate keys will become an error in future releases, and are errors
by default when using the new API.
`
Thanks in advance : )

There seems to be missing file in dataio_utils

The full_load_io doesn't implement the JSON loader necessary for io of sample data and squad's JSON format.

tokenid2charsid is also not implemented.

AtributeError: 'DataIO' object has no attribute 'tokenid2charsid'.

Unfortunately training doesn't work for me, I'm getting the following error (abbreviated):

use config: default_gpu I:[helper.py:188]:number of jobs in the queue: 1 I:[helper.py:190]:will start the job: python app.py train /tmp/daanet-sb_vvrih ... use config: default_gpu I:0:0915-124247:[hel:par:119]:loading parameters... I:0:0915-124247:[hel:par:128]:[add from /tmp/daanet-hfxaocgi] task_name: qag I:0:0915-124247:[hel:par:128]:[add from /tmp/daanet-hfxaocgi] train_files: ['./sample_data/sample.json'] I:0:0915-124247:[hel:par:128]:[add from /tmp/daanet-hfxaocgi] dev_files: ['./sample_data/sample.json'] I:0:0915-124247:[hel:par:128]:[add from /tmp/daanet-hfxaocgi] test_files: ['./sample_data/sample.json'] I:0:0915-124247:[hel:par:128]:[add from /tmp/daanet-hfxaocgi] word_embedding_files: ['./sample_data/sample.embed.txt'] I:0:0915-124247:[hel:par:128]:[add from /tmp/daanet-hfxaocgi] char_embedding_files: ['./sample_data/sample.charembed.txt'] I:0:0915-124247:[hel:par:128]:[add from /tmp/daanet-hfxaocgi] result_dir: ./sample_data/result Traceback (most recent call last): File "app.py", line 15, in <module> run() File "app.py", line 10, in run args = parse_args(sys.argv[2] if len(sys.argv) > 2 else None, MODEL_ID, CONFIG_SET, followup_args) File "/home/undef/readingComprehension/daanet/utils/helper.py", line 148, in parse_args shutil.copytree('./', hparams.get('code_dir'), ignore=shutil.ignore_patterns(*IGNORE_PATTERNS)) File "/home/undef/readingComprehension/daanet/env/lib/python3.6/shutil.py", line 359, in copytree raise Error(errors) shutil.Error: [('./save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020205/code/save/0915-020...

I'm interested in dual ask-answer model too much.
I weired error in below

File "/Users/sunu/IdeaProjects/QAZeroToAll/daanet/daanet/basic.py", line 169, in _embed
trainable=True)
File "/Users/sunu/IdeaProjects/QAZeroToAll/daanet/nlp/nn.py", line 91, in get_var
regularizer=regularizer_fn, **kwargs)
...
ValueError: You can only pass an initializer function that expects no arguments to its callable when the shape is not fully defined. The given initializer function expects the following args ['shape', 'dtype', 'partition_info']

I wondered why
daanet / basic.py
only use the get_var () method in self.char_emb

hi, how to run a code on cpu help me..

Can you please elaborate more for steps to train, test and evaluate. And things like where the data is stored, approximately how much time it takes for metrics evaluation. It would be really of a good help to everyone.

I was trying to run the code there seem to be multiple errors in utils.helper with HParams (TypeError: argument of type 'HParams' is not iterable) and the save folder gets created recursively with the entire code in it all over again multiple times. Could you please update the code with the necessary changes?

Thanks!

Can you include the code for a progress bar? This will help to know how much training is completed.