Questions about evaluating duplicate corrections about errant HOT 2 CLOSED

Thank you for your reply.
Now I understand that this behavior is reasonable from your kind explanation.

this looks like a one-in-a-million kind of edge case

I agree. We would almost never encounter such a case 😂

Thanks again!

from errant.

Heya! That's a good question. It looks like a case of ERRANT not merging something I would have hoped it would merge.

Specifically, it seems the human annotator wanted to add , or something , (note the insertion of 2 commas) into the reference to make it make sense with the reference to a place later in the sentence. I would have hoped ERRANT would group this together as a single multi-word insertion edit: e.g.

A 7 7|||M:OTHER|||, or something,|||REQUIRED|||-NONE-|||0

... but it instead chooses to split it into 4 separate insertion edits at the same place.

A 7 7|||M:PUNCT|||,|||REQUIRED|||-NONE-|||0
A 7 7|||M:CONJ|||or|||REQUIRED|||-NONE-|||0
A 7 7|||M:NOUN|||something|||REQUIRED|||-NONE-|||0
A 7 7|||M:PUNCT|||,|||REQUIRED|||-NONE-|||0

Since the hypothesis correctly inserted a comma after somebody however, it looks as though it matches both the comma at the start of the phrase and the comma at the end of the phrase. The fix would thus be to make sure the reference edits are not split into several smaller edits, but sadly, I know I can't do that without negatively affecting other edit alignments.

I agree that it doesn't make sense for the number of TPs to exceed the number of edits, but this looks like a one-in-a-million kind of edge case to me. The errant.Annotator.annotate() has actually output the correct number of edits from the reference, and if the hypothesis matched the reference, then it should be rewarded with 2 TPs for matching both commas.

In short - there's not a lot I can do about it other than say you found a needle in a haystack!

from errant.