jeniyat / wnut_2020_ner Goto Github PK
View Code? Open in Web Editor NEWThis repository will contain the data and codes for WNUT 2020 NER task
License: MIT License
This repository will contain the data and codes for WNUT 2020 NER task
License: MIT License
Hi! I was looking at the dataset and I found some differences in some of the counts when looking at the standoff vs the conll data. When you look at the number of sentence in the standoff training data it matches the number in your readme.
$ cat data/train_data/Standoff_Format/*.txt | wc -l
8436
$ cat data/dev_data/Standoff_Format/*.txt | wc -l
2838
$ cat data/test_data/Standoff_Format/*.txt | wc -l
2809
(Note that number of sentences in the standoff format test data don't seem to match your read me either unless the sentences in the read me aren't equivalent to steps in the protocol?)
However when I looked at the conll formatted data, first using my custom conll reading code and then using the conllu
package from pypi, I am seeing more sentences in the training and development data (8444 and 2839 respectively) and fewer sentences in the test data (2803)
Here is the code I used to get the counts.
from conllu import parse
from glob import iglob
docs = 0
sentences = 0
for file in glob("data/train_data/Conll_Format/*_conll.txt"):
docs += 1
with open(file) as f:
sents = parse(f.read(), fields=["suface", "NER"])
sentences += len(sents)
print(f"# of documents: {docs}")
print(f"# of sentences: {sentences}")
# of documents: 370
# of sentences: 8444
Any idea what is going on?
Hi! I was looking at the released data and code and noticed some things that led me to wonder: Is relation extraction still part of this Shared Task?
There are inconsistencies between the Standoff and the CoNLL data, making combining the data difficult. (A similar issue was already raised in #3.) These are related to tokenization. Two randomly chosen examples:
protocol_7.ann
, line 268, the token untransformed
actually contains a space at the end, which messes up token boundaries. This seems to be because it is a non-breaking space and happens multiple times in the data.protocol_102_conll.txt
, line 149, contains the token FLowCamMake
, labeled as B-Device
. In contrast, protocol_102.ann
correctly identifies that there is actually an entity boundary with in this sequence of characters, splitting it up into FLowCam
and Make sure
.Unless I am missing something, there does not seem to be an official evaluation script for relations/events. The Standoff-to-CoNLL conversion script also seems to completely ignore relations. The baseline system also seems to only predict entities.
The Readme.md is titled "WNUT 2020: Named Entity Extraction", not mentioning relations.
So my question is: Are relations still part of the Shared Task? If so, will an official evaluation script be released for them?
Predictions are converted to conll after prediction made. From line 224 to 231 in code/baseline_CRF/crf_ner_wlp.py, the format is converted into standoff, and again converted into conll. Could you plz explain why we need this double conversion?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.