spyysalo / conlleval.py Goto Github PK

The break in sequences where an E tag is followed by a B tag is ignored. For example,

North B-MISC B-MISC
African E-MISC I-MISC
Grand B-MISC I-MISC
Prix E-MISC E-MISC

(correct: "North African" and "Grand Prix", two mentions; predicted: "North African Grand Prix", one mention) is evaluated as

processed 4 tokens with 1 phrases; found: 1 phrases; correct: 1.
accuracy:  50.00%; precision: 100.00%; recall: 100.00%; FB1: 100.00
             MISC: precision: 100.00%; recall: 100.00%; FB1: 100.00  1

Giving a contradictory 100% f-score with 50% accuracy.

This occurs also in the original perl version. While not an issue for the IOB and IOE schemes that the script explicitly supports (see http://arxiv.org/pdf/cs/9907006.pdf), this is arguably wrong for IOBES (which mostly evaluates as expected).

My output.txt file is the following format:
Sao NC B-LOC B-LOC
Paulo VMI I-LOC I-LOC
( Fpa O O
Brasil NC B-LOC B-LOC
) Fpt O O
, Fc O O
23 Z O O
may NC O O
( Fpa O O
EFECOM NP B-ORG B-ORG
) Fpt O O
. Fp O O

Which should be okay according to the conll format. But after running the conll.py I am getting this error:
OSError: conlleval: unexpected number of features in line Sao NC B-LOC B-LOC
Do you have any idea why? @spyysalo
Thanks in advance!

IOBES tags should be supported (related to #1).

The script takes input as a sequence of strings in CoNLL format:

https://github.com/spyysalo/conlleval.py/blob/master/conlleval.py#L57

This is fine for command-line use, but cumbersome for calling from Python. There should be a way to invoke evaluation with Python objects.

h/t: @marekrei