About the MSR RTE dataset used about sapphire HOT 2 CLOSED

Hello @Punchwes,

I used the dataset downloaded from https://github.com/xuchen/jacana/tree/master/alignment-data/msr/converted instead of the original MSR RTE 2006 dataset link. These files are the dataset used in Yao et al. (2013) and are different from those used in Ouyang and McKeown (2019) and our paper.

The preprocess by Ouyang and McKeown (2019) (and our paper) is simple:

1. exclude sentence pairs with no possible alignment,
2. merge simply consecutive alignments, e.g. ['cute', 'cute'] and ['dog', 'dogs'] → ['cute dog', 'cute dogs'].

(On evaluation, they decomposed phrase alignments to word alignments, like ['cute dog', 'cute dogs'] → ['cute', 'cute'], ['cute', 'dogs'], ['dog', 'cute'], and ['dog', 'dogs']. See their paper for the reason for this process.)

I will possibly share the preprocess scripts and/or processed data later if needed and possible.

from sapphire.

Thanks very much! That is very helpful!!

from sapphire.