batch absent_exact P/R/F1/Corr @50 about opennmt-kpg-release HOT 4 CLOSED

In addition, I noticed that the F1@10 > F1 @5, while in your paper, F1@10 < F1@5

from opennmt-kpg-release.

Thank you for your questions. I cannot give any feedback without knowing your running arguments and dataset.

from opennmt-kpg-release.

Thanks for your reply!
I just run this command which is provided in README
python kp_gen_eval.py -tasks pred eval report -config config/test/config-test-keyphrase-one2seq.yml -data_dir data/keyphrase/meng17/ -ckpt_dir models/keyphrase/meng17-one2seq-kp20k-topmodels/ -output_dir output/meng17-one2seq-topbeam-selfterminating/meng17-one2many-beam10-maxlen40/ -testsets nus -gpu -1 --verbose --beam_size 10 --batch_size 32 --max_length 40 --onepass --beam_terminate topbeam
and got the results in the picture.
The F1@10 is 0.3772 while in the paper "Does order matters?" the result is 0.295.
I used your provided pre-trained models.
Thanks very much!

from opennmt-kpg-release.

The default search strategy is full as set in opts.py L670. This means it during beam search every search sequence will not be terminated until it reaches the max_len (say 50). It can boosts scores a lot when beam_size is small at the cost of slow searching speed. topbeam has merits of fast inference as well as comparable performance when beam_size is large enough. The F1 0.295 is based on the topbeam decoding. We will provide more comparisons about these two settings in the paper a little later.

from opennmt-kpg-release.