Which tokenizer did you use? about tagger HOT 11 CLOSED

Hi,

I only trained the model on the CoNLL datasets that were already tokenized, so I did not have to tokenized anything. Probably the Moses tokenizer should work well:
https://github.com/moses-smt/mosesdecoder/tree/master/scripts/tokenizer

from tagger.

@glample can you give one example line how the input.txt should look like?

from tagger.

Yes, you can check the data here:
https://github.com/glample/tagger/tree/master/dataset

from tagger.

@janwendt What did you eventually do?

from tagger.

@mrmotallebi I am using the StanfordCoreNLP API which does a very good job but there are similar Python libs (NLTK is pretty good) as well.
Post that got me to the API: https://www.ibm.com/developerworks/community/blogs/nlp/entry/tokenization?lang=en

from tagger.

I would personally recommend the Moses one, it's pretty standard, and very fast.

from tagger.

@janwendt Do you have a domo of input.txt to the tagger.py?

from tagger.

@janwendt It needn't . I have successed.

from tagger.

@bjtu-lucas-nlp Could you please share an example of input.txt?
I have tried all kinds of combination but still get O tags of everything in the output.

from tagger.

@gui-li if you have a column based data structure like:
example1;example2;example3;

the tokenized output and input.txt for your tagger should be:

example1
example2
example3

from tagger.

@janwendt Thanks for replying.

from tagger.