Problems in metrics about jointre HOT 3 CLOSED

Hello,

1. I followed Adel and Schutze (2017) to use macro-averaged F1, so that I could compare with their scores (You can ask them why they did not use micro-averaged F1). In addition, Bekoulis et al. (2018) also used macro-averaged F1.
2. I also followed Adel and Schutze (2017) to discard the "Other" entity type. I think Bekoulis et al. (2018) also discarded the "Other" entity type as here.

from jointre.

Thanks for your answer,

You are right, in setup 2 both (Adel and Schutze 2017) and (Bekoulis 2018) use macro F1, I was not aware of that.

But in setup 1 which is the true end-to-end RE, (Bekoulis 2018) and previous works use micro-F1 score.
The lines you point to are after if self.config.evaluation_method == "relaxed": which corresponds to your setup 2.

Otherwise he uses getChunkedOverallF1() from eval.py and you can see that it is micro-average and that "Other" is not discarded.

I agree that all these different settings are confusing but it is necessary to ensure that the comparisons are valid.

from jointre.

Bekoulis et al. (2018)'s paper did not clarify which F1 (micro/macro) they used. As they used Adel and Schutze (2017)'s data, I assumed they followed the macro-averaged metric used in Adel and Schutze (2017) for both setups.

Regarding Setup 1, if you think Bekoulis et al. (2018) used micro-averaged F1, you can compare to Bekoulis et al. (2018) w.r.t. micro-averaged F1. Also, you can additionally compute your macro-averaged F1 in comparison to jointER.

I believe that whatever metric used for Setup 1 does not affect the conclusion and contributions of our paper. We still obtain SOTA performance under Setup 2 which is used in most previous work on CoNLL04.

from jointre.