I try the same code with same compound string but the score is -0.65. Why is that so? e.g: "Autobahnraststätte" returns:
[-0.6561174724342663, 'Autobahn', 'Raststätte'],
[-0.719082070992539, 'Autobahnrast', 'Stätte'],
[-2.0207606162242056, 'Auto', 'Bahnraststätte'],
[-2.0883545770567786, 'Autobahnrasts', 'Tätte'],
[-2.116115029842648, 'Autobahnrastst', 'Ätte'],
[-2.1366906474820144, 'Autobahnras', 'Tstätte'],
[-2.155172413793103, 'Autobahnra', 'Ststätte'],
[-2.1557478368356, 'Autobahnr', 'Aststätte'],
[-2.237077877325982, 'Aut', 'Obahnraststätte'],
[-2.458303592671901, 'Autob', 'Ahnraststätte'],
[-2.709178455383891, 'Autobahnraststä', 'Tte'],
[-2.785514345696291, 'Autobah', 'Nraststätte'],
[-3, 'Autoba', 'Hnraststätte']]

This compound splitter is amazing thank you.
The problem I have with it is that it tends to split even words that it really shouldn't, for example the word "Präsident" is split into "Prasid", "ent".

This flaw makes it impossible for me to use in my NLP project.

I was wondering if you had a way to filter out words to be split and words not to be split.

This would be really helpful, thank you.

Thank you a lot for this great compount splitter. It works really good. Unfortunately it does have some problems with hyphens.

char_split.split_compound("Kraftfahrzeug-Haftpflichtversicherung")
returns [[1.0, 'Kraftfahrzeug-Haftpflichtversicherung', 'Haftpflichtversicherung']]

The whole word is returned in the first part of the result. Do you know how to fix this issue?
thanks in advance :)

Thank you for sharing the code. Your compound splitter works very well, however, maybe not as intended. For each split position n instead of

score(n) = max p(prefix) + max p(suffix) − min p(infix),

what is actually computed by the code is

score(n) = p(word[:n]) + p(word[n:]) - min p(infix).

That is, the code iterates over all infix ngrams, but only uses the full prefix and suffix for the other two probabilities. I simplified the code to reflect this and removed some redundant steps: https://github.com/kldtz/CharSplit. Now the GermaNet evaluation runs in about half of the original time on my machine (yielding the same result).

I also tried using all prefix and suffix ngrams as described in the appendix of your thesis, but this has a negative effect on the performance for the GermaNet evaluation set.

Thanks for sharnig your great repo.

Would it be possble for you to make available a pypi package for this repo?

Is it possible to find (or generate) the training data to experiment with algorithm?