About the second encoder for the relation model about pure HOT 3 CLOSED

Hi! Thanks for your interests! A "second pre-trained encoder" means that we use a separate encoder for the relation model (instead of sharing the encoder with the entity model). Both relation encoder and entity encoder are BERT.

Sorry about the confusion! Let me know if you have any further questions!

from pure.

Hello! Thanks for your help! But I still have a question about it. As you mentioned, the additional inserted marker plays an important role in obtaining the relation representation. Here is the thing. As BERT haven't seen these markers before, so I find it really hard to understand how to use BERT to get reasonable representation for these markers. I guess there are some pre-trained tasks like MLM to help BERT see these special markers at first? Please correct me if I miss the point.

Thanks,

from pure.

You are right that BERT haven't seen these markers during pre-training. We don't have any special pre-training tasks for marker tokens. The representations of marker tokens are learned merely during fine-tuning on the downstream task (relation classification).

from pure.