b0noi / aif2 Goto Github PK

to use AIF2 library in unit tests

So far we assume that all sentence split characters are end-sentence characters, this is good for Alpha1 but should be fixed in Alpha2

All splitters should have one same name (split?)

If there is some text with spaces in the beginning and "-" character after it. Empty string will be saved as first token. To check this issue use should_get_tokens_from_text_file_with_space_in_the_begining_using_PREDEFINED_separator test for TokenSplitter.

We need to separate RAW tests and Functional tests (http://testng.org/doc/index.html)

Everything on wiki =)

http://www.reddit.com/r/languagetechnology

Just a quality which shows how it work in each lib

org.apache.commons.lang3.ArrayUtils used in ProbabilityBasedTokenSeparatorExtractor
but commons-lang lib not included in dependencies

it will be used for a quality test etc.

we need to be consistent with world-wide terminology

Always return empty list

Words in the text placed on two lines of text without spaces betwing splitted as one token.

Precondition:

1. There is text placed in several lines without spaces after new line
2. tokenSplitter instance use PREDEFIND type of TokenSeparatorExtractor

Steps

1. Use tokenSplitter to split text

Expected result:

1. All words in the text splitted on separated token - OK

Actual result

1. All words merged in one token after split - NOT OK

For good naming example we can use: http://www.monlp.com/2012/03/13/segmenting-words-and-sentences/

we should't splitt: it's => it s

we should split ONLY if characters at the end of token

to use OpenNLP library in unit tests

to use Stanford library in unit tests

http://nlp.stanford.edu/software/corenlp.shtml

All splitters should have one same name (split?)

Collect texts for 3+ languages, in each languages: 3+ text with size 10-15k words.

Each text should include sentence counter in it