malllabiisc / embedkgqa Goto Github PK
View Code? Open in Web Editor NEWACL 2020: Improving Multi-hop Question Answering over Knowledge Graphs using Knowledge Base Embeddings
License: Apache License 2.0
ACL 2020: Improving Multi-hop Question Answering over Knowledge Graphs using Knowledge Base Embeddings
License: Apache License 2.0
Hi,
Thank you for amazing work.
I have trained MetaQA model for question answer. I got around 0.95
accuracy.
Now how can i run the inference on new questions?
Thank you.
I used your command, your data, KG embedding and the bn weights and I still get 0.678. Can you provided the log file or other advice to help me rise the accuracy?
python main.py --mode train --relation_dim 200 --hidden_dim 256
--gpu 2 --freeze 0 --batch_size 128 --validate_every 5 --hops 3 --lr 0.0005 --entdrop 0.1 --reldrop 0.2 --scoredrop 0.2
--decay 1.0 --model ComplEx --patience 5 --ls 0.0 --kg_type full
ps: I have used the LSTM
I have successfully run the training code but it gave no print out. And I cannot run the code in eval mode since there's no eval function. Can you give the eval code? Thx
In the MetaQA knowledge embedding, for example, a relation directed_by
and directed_by_reverse
both have a embedding, but in the Webquestion Freebase Embedding, your pretrainning model didn't have the reverse embedding. Is there some reason not to train that?
By using your code without your relation matching in half KG metaQA, I found the result is 0.68 in hop1, 0.51 in hop2, 0.45 in hop3, there is a big gap with your paper report (even without relation matching, relation matching is used to solve: the big distance between topic entity and answers). I do not undertand how to achieve your relation matching in half KG, can you report your result without relation matching? By your describtion, it's not easy to realise the relation matching model.
Can you supply the script about how to get the best parameters for your model? Includes: learning rate, batch_size, etc
I found your websqp kg embedding file is None, can you supply it? When I run your training embedding code with your KG (relevant your websqp ) . I found there is an error, CUDA out of memory
Can you suply the relevent embedding file, just like metaqa??
I can't find the eval function in roBERTa main.py Can I get this function?
When would you like to share the codeof relation score?
Best Wishes!
Hi,Man
I want to know the exactly environment the code was running !
For exmple,pytorch version? python version 3.4 3.5 or 3.6 ? Developing environment :Ubuntu or Mac ?or Windows ?
Please update a detailed description of the running environment for the code replay ~ Thanks a lot !@apoorvumang
Add a way to eval a pretrained model, independent from training. This will require auto-saving of a model.
Hi,
We have build KG by extracting entities and relationship between entity from PDF documents. But out KG is not so big, we have around 5k nodes.
Now we wanted to use EmbedKGQA source code to build Natural Language question on custom KG.
I have below 2 questions,
Thanks
The command i used is :
'python main.py --mode train --relation_dim 200 --do_batch_norm 0
--gpu 0 --freeze 1 --batch_size 16 --validate_every 10 --hops webqsp_full --lr 0.00002 --entdrop 0.0 --reldrop 0.0 --scoredrop 0.0
--decay 1.0 --model ComplEx --patience 20 --ls 0.0 --l3_reg 0.001 --nb_epochs 200 --outfile full_fbwq'
can you supply the half metaQA training command? to get the entity embedding and relation embedding?
When do you provide data and instructions?
Hello Dr. Apoorv,
According to your email, RelScore does not have a gradient. the scoring function S(r,q) has a gradient and that is trained separately. And RelScore is only used in valid and testing phase. I wanna know some details, do you mean S(r,q) induces a part of loss during training or it is trained after all training epoches?
how to run knowledge graph embedding?
how to run KBQA model?
the data and pretrained_models are both only 134 byte large and cannot be opened. Even though downloading the single file leads the same result. Do you know why it is, or can you give another way to download them? Thank you!
In KGQA/RoBERTa/main.py you import kge module. Although I've installed the kge module from pip, it shows "ModuleNotFoundError: No module named '_Box2D'", where there's even no even the same situation on google. Have you encountered it and how did you overcome?
You define a variable named "hidden2rel" at line 85, but you call the function "hidden2rel" at line 125 in function "applyNonLinear", where there's no definition of function "hidden2rel". Sorry for my poor coding level, can you tell me where's the definition of the function "hidden2rel" or what it means?
I've trained a 200-dim ComplEx embedding for the knowledge graph of MetaQA.
As the best hyperparameters are not given, I directly use the default parameters.
The EmbedKGQA model works well in 1hop QA and gets 0.957 accuracy which is not too far from the value 0.975 given in the paper.
However when I tried 2hop and 3hop QA, I got 0.09 and 0.326 respectively. It seems to be more than just a issue of hyperparameters.
I find that in the test set of 2hop and 3hop MetaQA, most triples (topic entity, some relation, answer entity) don't appear in the knowledge graph kb.txt ( 4.97MB ), especially 2hop dataset ( 14872 QAs' in test set ). In 2hop set there are only 8 questions which have link between topic entity and answer entity in kb.txt. Also, the overlap (topic entity, some question, answer entity) between test and train set is only a small fraction of test set (less than a half).
Am I using the wrong dataset? Did I miss some important details?
This situation is similar with that described in Section 5.4: there is no link between head entity and answer entity. I dont understand how it could be possible that got 0.299 accuracy in 1hop( Table 4 in your paper) but 0.988 in 2hop( Table 2 in your paper).
The program only takes up more than 1G of GPU memory, which is very fast when run alone. However, if there are other programs running on this GPU, it will also take up more than 1G of GPU memory, but the speed will become as slow as the CPU, why? If there are other programs that use the GPU's other memory while the program running, the speed will suddenly become very, very slow (about a few hundredths of a percent). What's going on?
Did the experiment on MetaQA in the paper use LSTM?
Dear authors,
In your code, the model rotate3 refers to Rotate which from "ROTATE: KNOWLEDGE GRAPH EMBEDDING BY RELA- TIONAL ROTATION IN COMPLEX SPACE"?
I want to do new Knowledge Graph training. I am curious about the parameter value
I have run the LSTM code for 3-hop on full KG settings, but I got a 0.7271963009667928 score. Do I need to change some parameters when I run the code for 3-hop?
Thanks!
Dear authors,
Why I get the 0.94 in hop2, 0.972 in hop1, 0.48 in hop3 (metaQA full KG), I used your code, same parameters, there is a big gap with the description in your paper, what should I do?
Can you see that the'WebQuestionsSP' executable code is running, but the'relation maching' code cannot be found?
Thx very much
Could you provide the setting of KG embedding modulo? i.e. how can I get E.py&R.py with your train_embedding code.
I cannot train it properly with default setting.
Hi, I found this paper very enlightening, taking full advantage of the nature of knowledge graph embedding to complete multi-hop QA.
However, I have some doubts while reading the paper on ACL 2020.
How are the head entities in a natural language question get extracted, did this paper assumes that there is only one head entity in the question?
Thanks.
I still get 0.49 in metaQA full KG-hop3. can you provided the whole project (KG embedding, command), or other advice? 0.49 is too low, I do not know what's wrong with it? I used your command, your data. I hop you can provided the relevant project again or other advice to help me rise the accuracy.
Sorry to bother you, I used command you provided in github (metaQA hop1 full-KG), I just get 0.93, while your paper is 0.97, there is a large gap, on the other hand, I found your paper uses ROBERT to obtian the question representation, while in your code, you use BI-LSTM. Can you explain these two questions? expecially one, why I get 0.93 by your parameter?
your command:
python3 main.py --mode train --relation_dim 200 --hidden_dim 256
--gpu 2 --freeze 0 --batch_size 128 --validate_every 5 --hops 2 --lr 0.0005 --entdrop 0.1 --reldrop 0.2 --scoredrop 0.2
--decay 1.0 --model ComplEx --patience 5 --ls 0.0 --kg_type half.
If this command is not the command of metaQA hop1 full-KG, can you supply the all command about (hop1,hop2, hop3) in full KG metaQA, WebQuestionsSP and all command about (hop1,hop2, hop3) in half KG, at last, the training command about how to get entity embedding in these two KGs, two half KGs.
Thanks! I hope your reply can help me!!!
Why you use states[0] to compute the question embedding in a batch_size? the size of states[0] is [batch_size, embedding_dim], can you explain what's the meaning about states[0] ? I think question_embedding should be state not state[0]. Can you help me?
def getQuestionEmbedding(self, question_tokenized, attention_mask):
roberta_last_hidden_states = self.roberta_model(question_tokenized, attention_mask=attention_mask)[0]
states = roberta_last_hidden_states.transpose(1,0)# torch.Size([64, 128, 768])
cls_embedding = states[0]# torch.Size([128, 768])
question_embedding = cls_embedding
I clone the project, but the files in it seems not work. In detail, it seems that the data.zip has been destroyed, and can not be open. Could you please help me solve the problem?
When do you plan to update the code?
Hope to see it soon.
Hi, thanks for providing your code!
I was wondering whether you have ever tried any other embeddings like RESCAL like you included in your 'model.py'.
Did you select 'ComplEx' because it achieves the best performance?
Hi, thank you for sharing the source code. Could you please explain how does the model find all candidate answers in answer selection module? Thank you.
How to eval the best model on the test set? Could you please give me the command?
Hello
Thanks for the material update.
However, it is not possible to decompress data.zip
I need 'WebQuestionsSP Data' in data.zip for the experiment
Could you please check?
Batch norm greatly improved training speed for MetaQA. Could you explain why you set do_batch_norm to 0 for WebQSP (lead to more training epoches and patience)? What if we set it to 1? And how much can RelScore improve the results?
Hi,
When reading your code I found there is not a part in KGQA/LSTM that you load the transformer and use the pre-trained embeddings for sentences, as what you have done in KGQA/RoBERTa. Is that true, and why?
Best,
Shuang
Thanks for opening source !
I have read the code, but I did not find the relation matching module. The current prediction logic seems to be predicting scores for all entities of KG. Will you provide the code for relation matching, because I am not sure how is the shortest path between head entity and candidate answer defined.
You've described that "Full dataset contains the original kb.txt as train.txt with duplicate triples removed", but what are valid.txt and test.txt at the same directory? Does KGE need to be verified and tested?
for i_batch, a in enumerate(loader):
model.zero_grad()
question = a[0].to(device)
sent_len = a[1].to(device)
positive_head = a[2].to(device)
positive_tail = a[3].to(device)
Since question represents the embedding of questions, sent_len represents the length of questions, positive_tail represents the lables(answers) of questions, what dose "positive_head" represent?
When will the code for the 'WebQuestionsSP' dataset be updated?
Best Regards
i found text "relation matching will have to be done separately."
how can i run relation matching?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.