Git Product home page Git Product logo

irnet's Introduction

IRNet

Code for our ACL'19 accepted paper: Towards Complex Text-to-SQL in Cross-Domain Database with Intermediate Representation

Environment Setup

  • Python3.6
  • Pytorch 0.4.0 or higher

Install Python dependency via pip install -r requirements.txt when the environment of Python and Pytorch is setup.

Running Code

Data preparation

  • Download Glove Embedding and put glove.42B.300d under ./data/ directory
  • Download Pretrained IRNet and put IRNet_pretrained.model under ./saved_model/ directory
  • Download preprocessed train/dev datasets from here and put train.json, dev.json and tables.json under ./data/ directory
Generating train/dev data by yourself

You could process the origin Spider Data by your own. Download and put train.json, dev.json and tables.json under ./data/ directory and follow the instruction on ./preprocess/

Training

Run train.sh to train IRNet.

sh train.sh [GPU_ID] [SAVE_FOLD]

Testing

Run eval.sh to eval IRNet.

sh eval.sh [GPU_ID] [OUTPUT_FOLD]

Evaluation

You could follow the general evaluation process in Spider Page

Results

Model Dev
Exact Set Match
Accuracy
Test
Exact Set Match
Accuracy
IRNet 53.2 46.7
IRNet+BERT(base) 61.9 54.7

Citation

If you use IRNet, please cite the following work.

@inproceedings{GuoIRNet2019,
  author={Jiaqi Guo and Zecheng Zhan and Yan Gao and Yan Xiao and Jian-Guang Lou and Ting Liu and Dongmei Zhang},
  title={Towards Complex Text-to-SQL in Cross-Domain Database with Intermediate Representation},
  booktitle={Proceeding of the 57th Annual Meeting of the Association for Computational Linguistics (ACL)},
  year={2019},
  organization={Association for Computational Linguistics}
}

Thanks

We would like to thank Tao Yu and Bo Pang for running evaluations on our submitted models. We are also grateful to the flexible semantic parser TranX that inspires our works.

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.microsoft.com.

When you submit a pull request, a CLA-bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repositories using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

irnet's People

Contributors

gaoyancheerup avatar jasperguo avatar microsoft-github-operations[bot] avatar microsoftopensource avatar substill avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

irnet's Issues

Preprocessing the spider dataset leads to different file sizes.

HI All,

I am currently trying to preprocess the provided basic spider data. After executing the data_preprocess.py and sql2SemQL.py files, the file size is 20,819KB which is less than the actual processed file (39,030KB) provided by the authors. I also get the same error message ("column * table error" ) on the column, for 72 queries.

Do you have any pointers on why do we get this error for 72 queries?

Thanks,
Anshu

Crashes without printing any error!

When I run the train.sh file then after 2-3 minutes without giving any error it exits. I have sufficient amount (16GB) of RAM. I have attached the screenshot. Does anyone faced the same issue? How can it be resolved? Tried on colab as well as local machine. Same issue is there on both.
Please help.
issue

error while training irnet-bert

@JasperGuo
can you give me the requirement for my system to train the irnet-bert smoothly? with python ,pytorch version and cuda (9.2 or 10)
my system spec are like below:
32gb ram
256gb ssd+1tb hdd
8 gb of 2080 NVIDIA gpu

Firstly i was getting this error [CUDA-OUT-OF-MEMORY]
error message

to solve this i have change batch_size to 8.
but now i got below error.

Screenshot from 2020-03-12 18-30-26

please help

i run python3 train.py and it runs for like 10 to 15 min and then gives me this error:

The size of tensor a (300) must match the size of tensor b (100) at non-singleton dimension 2
Traceback (most recent call last):
File "/mnt/c/Users/Legion/irnet/IRNet/train_test.py", line 56, in train
loss = utils.epoch_train(model, optimizer, args.batch_size, sql_data, table_data, args,
File "/mnt/c/Users/Legion/irnet/IRNet/src/utils.py", line 242, in epoch_train
score = model.forward(examples)
File "/mnt/c/Users/Legion/irnet/IRNet/src/models/model.py", line 215, in forward
table_embedding = table_embedding + tab_ctx
RuntimeError: The size of tensor a (300) must match the size of tensor b (100) at non-singleton dimension 2

Values in the filter field is discarded in the final result.

Hi, I tried to test the pre-train model with the original data from Spider. However, I notice that the final result generated by the sem2SQL.py replaces all the filter value by 1. For example when the question is: What is the average, minimum, and maximum age of all singers from France? The result is SELECT avg(T1.Age), min(T1.Age), max(T1.Age) FROM singer AS T1 WHERE T1.Country = 1 Instead of the correct SQL SELECT avg(age) , min(age) , max(age) FROM singer WHERE country = 'France'.

I notice in the transfer function, we put value to None when calling to_str. Just wondering, is there a way to keep the values in the filtering query?

Many thanks.

Alternative to IRNET

Hi Team,

Since IRNET launch there are couple of models which was proposed in spyder are working with better accuracy than this...But issue with most of them that they work on Data and not on schema...If anyone know better SQL predicting algorithm than IRNET which work better than IRNET

Issue in model.forward

Hello guys,
when i run this model for training, i got an error " The size of tensor a (300) must match the size of tensor b (100) at non-singleton dimension 2".
in this line of model.forward ==> table_embedding = table_embedding + tab_ctx
. could anyone please answer how to solve this issue ?

RuntimeError: Error(s) in loading state_dict for IRNet

HI All, I am getting below error while loading IRNET trained model, i am using the final end_model.model to evaluate

/python3.8/site-packages/torch/nn/modules/module.py", line 846, in load_state_dict
raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for IRNet:
size mismatch for lf_decoder_lstm.weight_ih: copying a param with shape torch.Size([1200, 556]) from checkpoint, the shape in current model is torch.Size([1200, 492]).
size mismatch for sketch_decoder_lstm.weight_ih: copying a param with shape torch.Size([1200, 556]) from checkpoint, the shape in current model is torch.Size([1200, 492]).
size mismatch for type_embed.weight: copying a param with shape torch.Size([10, 128]) from checkpoint, the shape in current model is torch.Size([10, 64]).
size mismatch for att_project.weight: copying a param with shape torch.Size([300, 428]) from checkpoint, the shape in current model is torch.Size([300, 364]).
Namespace(data_path='./spider', input_path='predict_lf.json', output_path='./results/irnet')

I have tried following :

  1. model.load_state_dict(pretrained_modeled,strict=False)

  2. from collections import OrderedDict
    new_state_dict = OrderedDict()
    #for k in pretrained_model.keys():
    for k,v in pretrained_model.items():
    #if k not in model.state_dict().keys():
    # del pretrained_modeled[k]
    print("actual key ::",k)
    new_state_dict[k]=v

    model.load_state_dict(new_state_dict)

Any suggestions would be helpful. Thanks

Confused about finding table of column *

I don't understand the inference process of finding table of column * , in the functiondef _parser_column0(self, sql, select).
Would you please give me some points.Thanks.

Need help to preprocess wikisql dataset to SemQL format?

HI ,
I am trying to convert wikisql dataset to SemQL format to test IRNET on other data sets. And test other models on the SEMQL representation of nl and sql query to justify if intermediate format can actually help.

Has anyone tried converting other datasets to SEMQL format to test IRNet on other datasets?
If not, can the authors please suggest how can i do so apart from manually mapping the json differences and try to convert to semql ?

Any help is appreciated.

Thanks and Best Regards
Anshu

Thank you for your work

Thank you so much for the style you perform in the codes. You did a really good job for coding.
Sincerely

mask with dtype torch.uint8, deprecated

I am getting alot of deprecation warnings with python 3.6 cuda 9.2 and pytorch 1.3
Any idea what I could do to avoid that?

/tmp/pip-req-build-vxpey3tb/aten/src/ATen/native/cuda/LegacyDefinitions.cpp:19: UserWarning: masked_fill_ received a mask with dtype torch.uint8, this behavior is now deprecated,please use a mask with dtype torch.bool instead.
/tmp/pip-req-build-vxpey3tb/aten/src/ATen/native/cuda/LegacyDefinitions.cpp:19: UserWarning: masked_fill_ received a mask with dtype torch.uint8, this behavior is now deprecated,please use a mask with dtype torch.bool instead.
^CTraceback (most recent call last):
File "train.py", line 117, in
train(args)
File "train.py", line 80, in train
sketch_loss_coefficient=args.sketch_loss_coefficient)
File "/home/eni/IRNet/src/utils.py", line 242, in epoch_train
score = model.forward(examples)
File "/home/eni/IRNet/src/models/model.py", line 281, in forward
pre_types.append(pre_type)
KeyboardInterrupt

Why is the lf_accuracy so low?

This is my train log(the front 17 epoch):
Epoch: 1, Loss: 12.740511, Acc: 0.032101, time: 123.496223
Epoch: 2, Loss: 6.106339,Acc: 0.050584, time: 134.791753
Epoch: 3, Loss: 4.475008, Acc: 0.050584, time: 153.214200
Epoch: 4, Loss: 3.618750, Acc: 0.055447, time: 168.053908
Epoch: 5, Loss: 3.046593, Acc: 0.065175, time: 182.702742
Epoch: 6, Loss: 2.570832, Acc: 0.066148, time: 198.103145
Epoch: 7, Loss: 2.241890, Acc: 0.061284, time: 210.114276
Epoch: 8, Loss: 1.898860, Acc: 0.060311, time: 239.237827
Epoch: 9, Loss: 1.692311, Acc: 0.062257, time: 241.019557
Epoch: 10, Loss: 1.526351, Acc: 0.069066, time: 278.005775
Epoch: 11, Loss: 1.324829, Acc: 0.072957, time: 286.218282
Epoch: 12, Loss: 1.185413, Acc: 0.071984, time: 276.896780
Epoch: 13, Loss: 1.124110, Acc: 0.067121, time: 284.550130
Epoch: 14, Loss: 1.017929, Acc: 0.059339, time: 303.098995
Epoch: 15, Loss: 0.913161, Acc: 0.066148, time: 321.235667
Epoch: 16, Loss: 0.824031, Acc: 0.060311, time: 341.108282
Epoch: 17, Loss: 0.755057, Acc: 0.061284, time: 356.403827
...
I hardly changed anything,what's wrong?

Preprocessing question: keys "col_set" and "names" in spider dataset

HI All,

I am trying to preprocess wikisql dataset to semql format. But i am not able to understand what does "col_set" and "names" correspond to?
How keys "col_set" and "names" get values from spider dataset?
This will help me in getting values for wikisql dataset.

Thanks
Anshu

taking very much time to run BERT embedding model

Hello, I implement pytorch bert embedding instead of glove embedding but it takes too long to complete even single epoch.

I implement bert 768 dimension model for this, but for very small dataset like 120 examples it takes 166 sec to complete one epoch while glove takes only 3 sec.

can anyone tell me why is it so and how to fixed it.
Thanks in advance!

Error occured on parseing SQL to semQL,like "SELECT COUNT(*) FROM ( SELECT T1.CountryId , ....) "

Hi.
When I parse SQL to semQL(sql2semQL.py), the SQL like"SELECT COUNT(*) FROM ( SELECT ... )",the rule_label is:

Root1(3) Root(5) Sel(0) N(0) A(3) C(0) T({'except': None, 'from': {'conds': [[False, 2, [0, [0, 3, False], None], [0, 9, False], None]], 'table_units': [['table_unit', 1], ['table_unit', 2]]}, 'groupBy': [[0, 3, False]], 'having': [[False, 3, [0, [3, 0, False], None], 2.0, None]], 'intersect': None, 'limit': None, 'orderBy': [], 'select': [False, [[0, [0, [0, 3, False], None]], [3, [0, [0, 0, False], None]]]], 'union': None, 'where': []})

So, I downloaded your preprocessed data, it seemed as above.

how to understand the vector 'col hot type'?

how to understand the vector 'col hot type'?
maybe it is an useful information, but I don't understand how it relates to the corresponding column?
How did you come up with this representation, is it a simple and efficient way?
Looking forward to your answer,thanks.

'What are the official names of cities that have hosted more than one competition?'

src_sent = [['what'], ['are'], ['column', 'official', 'name'], ['of'], ['table', 'city'], ['that'], ['have'], ['hosted'], ['more'], ['than'], ['one'], ['competition'], ['?']]

tab_cols = [['count', 'number', 'many'], ['city', 'id'], ['official', 'name'], ['status'], ['area', 'km', '2'], ['population'], ['census', 'ranking'], ['farm', 'id'], ['year'], ['total', 'horse'], ['working', 'horse'], ['total', 'cattle'], ['ox'], ['bull'], ['cow'], ['pig'], ['sheep', 'and', 'goat'], ['competition', 'id'], ['theme'], ['host', 'city', 'id'], ['host'], ['rank']]

col_hot_type = [[0. 0. 0. 0.], [1. 0. 0. 0.], [2. 5. 0. 0.], [0. 0. 0. 0.], [0. 0. 0. 0.], [0. 0. 0. 0.], [0. 0. 0. 0.], [0. 0. 0. 0.], [0. 0. 0. 0.], [0. 0. 0. 0.], [0. 0. 0. 0.], [0. 0. 0. 0.], [0. 0. 0. 0.], [0. 0. 0. 0.], [0. 0. 0. 0.], [0. 0. 0. 0.], [0. 0. 0. 0.], [1. 0. 0. 0.], [0. 0. 0. 0.], [1. 0. 0. 0.], [0. 0. 0. 0.], [0. 0. 0. 0.]]

any information for 'one_hot_type'

I was reading the code in the 'model' part. The class 'example' contains one_hot_type to describe
the preprocessed question. But I couldn't find its use in modeling part.

why do you initialize decoder init state like '[tanh(linear(enc_last_cell_state)), zeros]'

I notice that in src/models/model.py you initialize the decoder init state like following:

    def init_decoder_state(self, enc_last_cell):
        h_0 = self.decoder_cell_init(enc_last_cell)
        h_0 = F.tanh(h_0)

        return h_0, Variable(self.new_tensor(h_0.size()).zero_())

It seams like the last cell state of question encoder is used as the initialized hidden state of the decoder after Linear() and tanh(), and a zero tensor is used as the initialized cell state of the decoder.

May I know the reason that you didn't use encoder's last (hidden state, cell state) as the initialization of the decoder's (hidden state, cell state) respectively?

How to convert sql to SemQL format to test IRNET for wikisql

Hi All,

I am stuck at parsing sql query of wikisql to irnet sql format especially for "sql" key :
for instance,
WIKISQL -> "sql":{"sel":5,"conds":[[3,0,"SOUTH AUSTRALIA"]],"agg":0}
IRNET SEMQL ->
"sql": {
"except": null,
"from": {
"conds": [],
"table_units": [
[
"table_unit",
1
]
]
},
"groupBy": [],
"having": [],
"intersect": null,
"limit": null,
"orderBy": [],
"select": [
false,
[
[
3,
[
0,
[
0,
0,
false
],
null
]
]
]
],
"union": null,
"where": []
}
}

Any help is appreciated.

Thanks
Anshu

Issues with preprocessing the raw spider data for IRNet

Hello all,

I am having some difficulties preprocessing the raw spider data using the preprocessing scripts provided with irnet. After executing the run_me.sh file for preprocessing, I am getting an error message ("column * table error" ) for 72 queries but it also outputs 'Finished 7000 datas and failed 0 datas' message which is confusing me.

When I am looking at the preprocessed train.json (output) file and comparing it with the preprocessed train.json file provided by the authors, I could see a lot of data (values/ keys) missing like 'truth_label', 'stanford_dependencies', 'true_label' etc. for each database and hence there is also a difference in their file size.
When I am training with my preprocessed file, I am getting 0.0 Accuracy which indicates there are some issues with the data files.

Any help or pointers would be appreciated.

Thanks,
Kanishk

json key "COL_PRED" not found

Hi @JasperGuo,

thanks for uploading bert base code,it ran perfectly for the already preprocessed dataset for spider. but there seems to be a problem in the preprocess code becuase when i tried to preprocess the data myself i found that a new key in the json "col_pred" is missing, it isn't getting genrated by the preprocess code. can you please look into at the earliest.

Is IRNet_pretrained.model supposed to achieve 50%+ dev accuracy?

I evaluate using eval.sh with IRNet_pretrained.model, and run spider official script. But I got strange result.

                     easy                 medium               hard                 extra                all
count                249                  438                  171                  170                  1028

====================== EXACT MATCHING ACCURACY =====================
exact match          0.084                0.078                0.088                0.065                0.079

---------------------PARTIAL MATCHING ACCURACY----------------------
select               0.207                0.204                0.340                0.253                0.234
select(no AGG)       0.224                0.209                0.346                0.259                0.243
where                0.175                0.168                0.111                0.176                0.162
where(no OP)         0.200                0.168                0.125                0.284                0.189
group(no Having)     0.091                0.318                0.286                0.333                0.285
group                0.000                0.217                0.122                0.286                0.182
order                0.000                0.105                0.275                0.298                0.164
and/or               1.000                0.912                0.898                0.890                0.927
IUEN                 0.000                0.000                0.105                0.148                0.071
keywords             0.369                0.313                0.246                0.224                0.298
---------------------- PARTIAL MATCHING RECALL ----------------------
select               0.201                0.192                0.304                0.235                0.220
select(no AGG)       0.217                0.196                0.310                0.241                0.228
where                0.194                0.158                0.090                0.133                0.148
where(no OP)         0.222                0.158                0.101                0.214                0.174
group(no Having)     0.150                0.315                0.359                0.177                0.269
group                0.000                0.215                0.154                0.152                0.172
order                0.000                0.133                0.186                0.173                0.148
and/or               0.936                0.960                0.961                0.942                0.951
IUEN                 0.000                0.000                0.051                0.111                0.080
keywords             0.433                0.303                0.205                0.188                0.283
---------------------- PARTIAL MATCHING F1 --------------------------
select               0.204                0.198                0.321                0.244                0.227
select(no AGG)       0.220                0.202                0.327                0.250                0.235
where                0.184                0.163                0.099                0.151                0.155
where(no OP)         0.211                0.163                0.112                0.244                0.181
group(no Having)     0.113                0.317                0.318                0.231                0.276
group                1.000                0.216                0.136                0.198                0.177
order                1.000                0.118                0.222                0.219                0.155
and/or               0.967                0.936                0.928                0.915                0.939
IUEN                 1.000                1.000                0.069                0.127                0.075
keywords             0.399                0.308                0.224                0.204                0.290

Did I do something wrong?
Thanks!

BTW, the length of prediction of IRNet is 1028, and the length of official dev_gold.sql is 1034.

I was wondering how can I get 53.2% accuracy like you.

Hello,
I repeat serval times the training and evaluation steps, but I can only get 52.89% accuracy in 1034 dev examples.

So I was wondering how can I get 53.2% accuracy like you. My trained model file is smaller (17M) than yours (23M). Why the size is different?

Thank you for your help.

Not reaching the ~53.4% on the Dev-Set

First of all, thanks for making this project public! I think it helps the community a lot to takle the problem of Text2SQL and it would be nice if other participants of the Spider-Challenge would do the same.

If I run this code with the parameters from train.sh, I reach an acc of close to 50% on the dev-set (see exact results below). Do I miss something? Or do I need to fix first the issues you talk about in #14 to reach this numbers? And if so, could you get a bit more specific in the changes?

Results:

Epoch: 0    Sketch-Accuracy: 0.5593385214007782     Accuracy: 0.2616731517509728
Epoch: 1    Sketch-Accuracy: 0.6196498054474708     Accuracy: 0.321011673151751
Epoch: 2    Sketch-Accuracy: 0.6955252918287937     Accuracy: 0.3949416342412451
Epoch: 3    Sketch-Accuracy: 0.7130350194552529     Accuracy: 0.40077821011673154
Epoch: 4    Sketch-Accuracy: 0.7422178988326849     Accuracy: 0.4377431906614786
Epoch: 5    Sketch-Accuracy: 0.7431906614785992     Accuracy: 0.4591439688715953
Epoch: 6    Sketch-Accuracy: 0.7431906614785992     Accuracy: 0.45525291828793774
Epoch: 7    Sketch-Accuracy: 0.7130350194552529     Accuracy: 0.42509727626459143
Epoch: 8    Sketch-Accuracy: 0.7558365758754864     Accuracy: 0.4581712062256809
Epoch: 9    Sketch-Accuracy: 0.7315175097276264     Accuracy: 0.42996108949416345
Epoch: 10    Sketch-Accuracy: 0.7558365758754864     Accuracy: 0.4698443579766537
Epoch: 11    Sketch-Accuracy: 0.7490272373540856     Accuracy: 0.4581712062256809
Epoch: 12    Sketch-Accuracy: 0.7665369649805448     Accuracy: 0.47373540856031127
Epoch: 13    Sketch-Accuracy: 0.7519455252918288     Accuracy: 0.45233463035019456
Epoch: 14    Sketch-Accuracy: 0.7402723735408561     Accuracy: 0.4766536964980545
Epoch: 15    Sketch-Accuracy: 0.7509727626459144     Accuracy: 0.4494163424124514
Epoch: 16    Sketch-Accuracy: 0.7422178988326849     Accuracy: 0.45622568093385213
Epoch: 17    Sketch-Accuracy: 0.7490272373540856     Accuracy: 0.4406614785992218
Epoch: 18    Sketch-Accuracy: 0.7159533073929961     Accuracy: 0.433852140077821
Epoch: 19    Sketch-Accuracy: 0.7538910505836576     Accuracy: 0.4406614785992218
Epoch: 20    Sketch-Accuracy: 0.7801556420233463     Accuracy: 0.4698443579766537
Epoch: 21    Sketch-Accuracy: 0.7665369649805448     Accuracy: 0.4785992217898833
Epoch: 22    Sketch-Accuracy: 0.7626459143968871     Accuracy: 0.46303501945525294
Epoch: 23    Sketch-Accuracy: 0.7558365758754864     Accuracy: 0.4785992217898833
Epoch: 24    Sketch-Accuracy: 0.7431906614785992     Accuracy: 0.44260700389105057
Epoch: 25    Sketch-Accuracy: 0.7509727626459144     Accuracy: 0.4785992217898833
Epoch: 26    Sketch-Accuracy: 0.7529182879377432     Accuracy: 0.47373540856031127
Epoch: 27    Sketch-Accuracy: 0.7675097276264592     Accuracy: 0.48151750972762647
Epoch: 28    Sketch-Accuracy: 0.7568093385214008     Accuracy: 0.4805447470817121
Epoch: 29    Sketch-Accuracy: 0.7636186770428015     Accuracy: 0.4727626459143969
Epoch: 30    Sketch-Accuracy: 0.7665369649805448     Accuracy: 0.47568093385214005
Epoch: 31    Sketch-Accuracy: 0.7558365758754864     Accuracy: 0.48151750972762647
Epoch: 32    Sketch-Accuracy: 0.7616731517509727     Accuracy: 0.46303501945525294
Epoch: 33    Sketch-Accuracy: 0.7704280155642024     Accuracy: 0.4727626459143969
Epoch: 34    Sketch-Accuracy: 0.7645914396887159     Accuracy: 0.4776264591439689
Epoch: 35    Sketch-Accuracy: 0.7568093385214008     Accuracy: 0.4698443579766537
Epoch: 36    Sketch-Accuracy: 0.7616731517509727     Accuracy: 0.48249027237354086
Epoch: 37    Sketch-Accuracy: 0.7665369649805448     Accuracy: 0.4571984435797665
Epoch: 38    Sketch-Accuracy: 0.7607003891050583     Accuracy: 0.45428015564202334
Epoch: 39    Sketch-Accuracy: 0.7597276264591439     Accuracy: 0.4581712062256809
Epoch: 40    Sketch-Accuracy: 0.7626459143968871     Accuracy: 0.4678988326848249
Epoch: 41    Sketch-Accuracy: 0.7655642023346303     Accuracy: 0.4688715953307393
Epoch: 42    Sketch-Accuracy: 0.7626459143968871     Accuracy: 0.4659533073929961
Epoch: 43    Sketch-Accuracy: 0.7665369649805448     Accuracy: 0.46303501945525294
Epoch: 44    Sketch-Accuracy: 0.7616731517509727     Accuracy: 0.4659533073929961
Epoch: 45    Sketch-Accuracy: 0.7568093385214008     Accuracy: 0.4669260700389105

what is meant by save_fold/ output_fold

I would like to know a suggestion for an average best practice [save_fold]/[output_fold] value.
I do not really understand what is meant by this option and how to understand the print statement

 $ sh eval.sh 0 100
 1028 0

Much Thanks

Cannot reproduce the IRNet+BERT results

Thank you for sharing the IRNet+bert model. The code is very useful, but I can not reproduce the results from the paper (61.9% dev acc).

Testing the pre-trained model downloaded from #2, I got Acc: 0.572957, Beam Acc: 0.598249 using your eval and 0.572 EM accuracy using the official evaluation script.

Testing the model trained from scratch (bert branch), I got similar results: Acc: 0.564202, Beam Acc: 0.600195 using your eval and 0.573 EM accuracy using the official script.

Do you have similar results from eval.py script? What is the accuracy of the released model?

getting loss as 'nan' after 1st epoch only?

Hello guys,
just like the Glove, I created a dictionary of all the possible words. with keys as words and values as 768 embedding vector for BERT.
But when I use this dictionary and train the model, the loss is getting nan in 1st epoch only.

  1. How to handle this problem?
  2. what are the possible reasons for getting a loss 'nan'?
  3. Is this a good approach, to make a dictionary of embedding vectors?

Trained Model

Hi, I am really curious to test your work, but do not have the necessary computational power on my local machine. My question is whether you can give access/share trained model, performing as good as your metrics state, or, possibly, you grant access to some API where model can be tested with user's tables and queries. Thank you

Is there any effect on accuracy as number of columns and tables in one database increases?

I have added one more database other than a spider and trying to do prediction with my own queries that database is AdventureWorks which is popular and Big, It has 71 tables and 400+ columns overall and have many to many relationships among tables, Now When I incorporated that My train accuracy is going down to 10% only. Almost all queries are failing. Is there any limitation of IRNet in terms of database table/column size?

In spider, there are these databases having limited numbers of tables/columns. I am curious to know if anyone has tried with their own databases which are large and have some insights

lr_scheduler.step() before optimizer.step() Error while executing the code

HI,

I am getting error while executing irnet on colab. I was able to run IRNET+bert on colab but not with glove. I also tried downgrading to the older version of pytorch, but still the execution of train.py stops without any warning

\lr_scheduler.py:123: UserWarning: Detected call of lr_scheduler.step() before optimizer.step(). In PyTorch 1.1.0 and later, you should call them in the opposite order: optimizer.step() before lr_scheduler.step(). Failure to do this will result in PyTorch skipping the first value of the learning rate schedule. See more details at https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate
"https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate", UserWarning)

Thanks for your help.
Anshu

IRNet cannot run after update

Hi, I downloaded IRNet's last version and ran it . However, it has a running error like this:
image
it seems caused by pytorch version, and i updated it to torch 1.4, and then it has another error:
image
how can we solve this error ?

thank you for your help

RuntimeError: generator raised StopIteration

Hi, I am getting the below error during testing :

Sketch Acc: 0.777237, Acc: 0.499027
Traceback (most recent call last):
File "C:\Users..\Anaconda3\lib\site-packages\pattern\text_init_.py", line 609, in _read
raise StopIteration
StopIteration

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "sem2SQL.py", line 671, in
alter_not_in(datas, schemas=schemas)
File "..\IRNET microsoft\src\rule\sem_utils.py", line 137, in alter_not_in
h_table = find_table(cur_table, origin_table_names, question_arg_type, question_arg)
File "..\IRNET microsoft\src\rule\sem_utils.py", line 105, in find_table
if partial_match(question_arg[i], table_name) is True and t_id != h_table:
File "..\IRNET microsoft\src\rule\sem_utils.py", line 33, in partial_match
query = [lemma(x) for x in query]
File "..\IRNET microsoft\src\rule\sem_utils.py", line 33, in
query = [lemma(x) for x in query]
File "C:\Users..\Anaconda3\lib\site-packages\pattern\text_init_.py", line 2172, in lemma
self.load()
File "C:\Users..\Anaconda3\lib\site-packages\pattern\text_init_.py", line 2127, in load
for v in _read(self._path):
RuntimeError: generator raised StopIteration

How can we fix this error ?

Thanks

IRNet+Bert版本

 您好,我在原有IRNet网络上加入Bert后,sketch acc可以上升到81,但SemQL  acc降到38,您觉得是什么原因吗?是因为参数问题吗,学习率,bacth size等吗
期待您的答复。

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.