ibalazevic / tucker Goto Github PK

Reverse relation is usually removed from datasets, but you add it to training data? What is the motivation for this?

In code

self.valid_data = self.load_data(data_dir, "valid", reverse=reverse)
self.test_data = self.load_data(data_dir, "test", reverse=reverse)

it should be

self.valid_data = self.load_data(data_dir, "valid", reverse=False)
self.test_data = self.load_data(data_dir, "test", reverse=False)

I did testing with this and the results it shows are much better than reported in the paper. Please let me know if I am wrong.

I used torch==1.0.0 and cannot reproduce the results in the paper. What different from torch==0.4.0 to torch==1.0.0? Thanks for your help!

Hi, thanks for your elegent code and job!!
I am thinking how to achieve the 1-x socre funtion( The x of 1-x means the number of entity to form a loss. 1-N uses the whole entity.）. In my opinion, 1-x should has much better performace than 1-N, becase is hard to train in high dim.
So why not use 1-x score function ?

Hi Ivana

I am trying to get entity embeddings for a downstream application. For WN18RR dataset I was unable to reproduce the reported results of TuckER. I used the hyperparameters given in the README of this repo. Following is the command I used:

 CUDA_VISIBLE_DEVICES=3 python main.py --dataset WN18RR --num_iterations 500 --batch_size 128 \
                                       --lr 0.01 --dr 1.0 --edim 200 --rdim 30 --input_dropout 0.2 \
                                       --hidden_dropout1 0.2 --hidden_dropout2 0.3 --label_smoothing 0.1

And the results are:

495
12.792492151260376
0.00035594542557143403
Validation:
Number of data points: 6068
Hits @10: 0.5121951219512195
Hits @3: 0.4728081740276862
Hits @1: 0.43638760711931446
Mean rank: 6254.662491760053
Mean reciprocal rank: 0.4624483298017613
Test:
Number of data points: 6268
Hits @10: 0.5140395660497766
Hits @3: 0.4738353541799617
Hits @1: 0.43123803446075304
Mean rank: 6595.924856413529
Mean reciprocal rank: 0.45961590280892123
5.328977823257446

Should I increase the number of epochs or am I missing something?

Thanks

Can you provide the parameters for reproducing the results from the paper on FB15k and FB15K-237? I ran the command from the README:

 CUDA_VISIBLE_DEVICES=0 python main.py --dataset FB15k-237 --num_iterations 500 --batch_size 128
                                       --lr 0.0005 --dr 1.0 --edim 200 --rdim 200 --input_dropout 0.3 
                                       --hidden_dropout1 0.4 --hidden_dropout2 0.5 --label_smoothing 0.1

which gave final performance of

Number of data points: 35070
Hits @10: 0.4009124607927003
Hits @3: 0.2555460507556316
Hits @1: 0.1760193897918449
Mean rank: 291.46401482748786
Mean reciprocal rank: 0.24741750020439274
Test:
Number of data points: 40932
Hits @10: 0.3974396560148539
Hits @3: 0.2546662757744552
Hits @1: 0.17094205022964917
Mean rank: 304.61949086289457
Mean reciprocal rank: 0.24344486414937788

Any ideas?

UPDATE: I noticed in the paper that you mention the best learning rate for FB15k-237 is 0.005 instead of 0.0005 and best the learning rate decay is 0.995 instead of 1.0 -- might that be the issue?

sort_idxs is numpy and e2_idx is torch
I changed the line 82 in main.py like below , and it worked
rank = np.where(sort_idxs[j]==e2_idx.cpu().numpy()[j])[0][0]

I am a bit confused about the evaluation protocol. In the evaluation, you only feed (head, rel) to the model and get predictions with n elements representing the scores of (head, rel, t_1) ... (head, rel, t_n). Why you don't repeat this process for the tail? Could you explain the reason? I think it should be done right? Previous works all conduct the evaluation in this way.
Maybe I misunderstand your code. Look forward to your reply.

Hi~
I have found that the code set "padding_idx=0" in nn.Embedding, like
self.E = torch.nn.Embedding(len(d.entities), d1, )
self.R = torch.nn.Embedding(len(d.relations), d2, padding_idx=0)
However, this will lead the gradient of the first entity and relation becoming zero. This is very interesting and I want to know the reason for this. Thank you!

Hi,

I have a question on the evaluation in the code.

when the test rank is evaluated, the scores seem only be calculated for each head toward all tails. I didn't see the scores are calculated for each tail toward all heads in the code. Don't people usually calculate them both and average them as the final scores? Would this one-way evaluation be a problem, such as having some bias?

Thank you!

Hey,
Thanks for releasing this. Could you clarify the license of the repo, couldn't find it anywhere in the Readme/code.

Is it MIT?

The default value for data_dir should be data/FB15k-237/ and not data/FB15K-237/ (lowercase the "K").

This paper mentions that when using realistic ranking, Tucker's performance decreases.

I was thinking if it was because of using the sigmoid before ranking which maybe causing numerical issues. It would be great if you could let me know what you think of this realistic ranking metric evaluation scheme!

I want to ask if your paper was published in journals or conferences.

def load_data(self, data_dir, data_type="train", reverse=False):
with open("%s%s.txt" % (data_dir, data_type), "r") as f:
data = f.read().strip().split("\n")
data = [i.split("") for i in data]
if reverse:
data += [[i[2], i[1]+"_reverse", i[0]] for i in data]
return data

Are your sure the data is data = [i.split("") for i in data] not data = [i.split("\t") for i in data], your data is splited by "\t", but you used space, if I use data = [i.split("\t") for i in data], I can not get the result you report in your paper about FB15k-237, can you explain it?

You have said that 'The RotatE (Sun et al., 2019) results are reported without their self-adversarial negative sampling (see Appendix H in the original paper) for fair comparison'

What is the reason for it being unfair?

In the paper, you say

for a given triple, we generate 2*n_e test triples by

1. keeping the subject entity e_s and relation r fixed and replacing the object entity e_o with all possible entities E and by
2. keeping the object entity e_o and relation r fixed and replacing the subject entity e_s with all entities E.

In the evaluate function, it looks like you score all possilbee_o's given an (e_s, e_r) tuple, then compute the rank of the true e_o. So I see how you're doing 1) above, but are you actually doing 2)?

Thanks!
~ Ben

Hey, I ran the code with suggested parameters, however, I was not able to reproduce the results on FB15k.

On FB15K, I got the following MRR (The best MRR is 0.789 in 500 epochs):
500
30.736143827438354
0.00022165691843464258
Validation:
Number of data points: 100000
Hits @10: 0.88763
Hits @3: 0.8288
Hits @1: 0.73175
Mean rank: 39.8066
Mean reciprocal rank: 0.789614621151087
Test:
Number of data points: 118142
Hits @10: 0.8898613532867228
Hits @3: 0.8294086776929458
Hits @1: 0.7293595842291479
Mean rank: 38.221682382218006
Mean reciprocal rank: 0.7889229105464421

Hi, thanks for developing this amazing model.
I'd like to try and train it on the Yago3-10 dataset (I think you have used it in your other work titled "Hypernetwork Knowledge Graph Embeddings").

Have you ever tried to train TuckER on that dataset?
Can you suggest me any hyperparameter settings, before I start running a long grid search? :)

Thanks for your help!

Andrea

Hi,

I was just going through your code and found out that the training data has been augmented by adding new relations for reversed triples from the training set (correct me if I am wrong). I am not sure whether this is harmless, as this might have a regularzing effect on the weights the model learns.

Instead of adding new relations for reversing the triples, could you try the following and check whether this gives the same result?

1. Create d.train_data_reversed, where for each triple from d.train_data you only switch e_s and e_o and keep the relation. (So you don't create any new relations in this dataset.)
2. Add to class TuckER a method forward_reversed that is exactly the same as forward, but transposes the tensor W, so that the axes for e_s and e_o are switched.
3. When training, use forward for d.train_data and use forward_reversed for d.train_data_reversed

I think this way, one can guarantee that the evaluation is fair. It would be also interesting to know how you evaluate other models you compare with, for examples, whether you use the BCE loss and augment the training data for other models as well. This will make sure that it is is not the BCE loss or data augmentation that helps TuckER perform well.

Hey if I just change the line 194 (d = Data(data_dir=data_dir, reverse=True) in main.py file, and use reverse=False, and run the code for FB15k-237 with recommended settings, the MRR shoots up to 0.4067. Is it expected behaviour?

To replicate:

1. Just change reverse=False in main.py
2. CUDA_VISIBLE_DEVICES=0 python main.py --dataset FB15k-237 --num_iterations 500 --batch_size 128 --lr 0.0005 --dr 1.0 --edim 200 --rdim 200 --input_dropout 0.3 --hidden_dropout1 0.4 --hidden_dropout2 0.5 --label_smoothing 0.1

MRR keeps increasing.

Log for iteration 145:
145
21.162700176239014
0.001321860825107114
Test:
Number of data points: 20466
Hits @10: 0.6135053259063813
Hits @3: 0.4763998827323366
Hits @1: 0.3409557314570507
Mean rank: 147.47825662073683
Mean reciprocal rank: 0.4335430118109515

Dear Ivana,

firstly thank you for the work and clean implementation. As I am interesting in relation prediction via TuckER, I was wondering whether you could provide a function like def forward_rel(self, e1_idx, e2_id): so that one could make use of TuckEr in relation prediction task.

Cheers

Hi , I experiment with your code and data on GPU, the parameters and settings are the same as yours, however ,it runs very slowly，Memory-Usage is 740M/12202M, my pytorch version is 0.4. What's the speed of your experiment? How to Improve？