Sentence segmentation about solrtexttagger HOT 7 OPEN

Could be language specific, genre specific, etc.

I don't think the tagger can deal with this, unless you are willing to take as an input the set of sentence spans. Otherwise, you being the tagger and one component of a larger pipeline would be dictating such segmentation. If you provide the option, you should expose it fully:

• allow caller to send in segment info
• allow caller to turn on SolrTextTagger segmenter, with option to report detected segments

otherwise it would seem half-baked to be making such decisions internal to tagger.

-m

from solrtexttagger.

I think that if some other component of a larger pipeline wanted to do segmentation, perhaps because it's more sophisticated, then the feature I propose here simply wouldn't be used. The BreakIterator based segmenter is better than nothing for those that don't have the time/expertise to develop the language specific, genre(?) specific segmenter.

What is your thoughts on breaking on all punctuation that's next to whitespace? It would have to be configurable of course, like perhaps not breaking on a hyphen.

from solrtexttagger.

Did you check that this actually happens? I just tried this sentence "All new. Norfolk gets a facelift" (because there is a New Norfolk (id:2155415) in the cities list) and it returns the following result (reformatted to return ("id", "name", "startOffset", "endOffset", "matchedSubstring")).

4776222 Norfolk 9 16 Norfolk
4945453 Norfolk 9 16 Norfolk
5073965 Norfolk 9 16 Norfolk
5128862 Norfolk 9 16 Norfolk

Assuming it does, we do something in user-land that might be of interest...

Sometimes we need to annotate at the sentence level, and doing a sentence per call works out to be too expensive (network overhead). So we create synthetic docs, say consider a max sentence size of maybe 500 chars and then join a bunch of sentences (say 50 or so) each space padded to 500 chars. Since the starting offset for each sentence is known, we can identify annotations based on the start and end offsets reported and also get the matching text from each sentence. We do this for performance, but one could use the technique to do sentence segmentation outside SolrTextTagger as well.

from solrtexttagger.

I'm sure "this can happen"; I could add a trivial test.

Nice trick on the sentence alignment. What could be useful is an additional TokenFilter that recognizes a large jump in startOffset, and if so increments the positionIncrement attribute, thus thwarting a match across the large span. If the Tagger sees a position increment > 1 (and if ignoreStopwords is false), it breaks any tags in progress (line ~128 of Tagger.java). ignoreStopwords has a dynamic default dependent on the presence of a StopWordFilter on the index analysis chain on the field.

from solrtexttagger.

In trying to reduce network overhead for tagging multiple separate entities (separate text fields in our case) in a single call, we did something similar ( see #35 ) , inserting a junk token which would never be matched between the concatenated entities, then using the returned offsets to demultiplex the returned tagger matches.

from solrtexttagger.

Great suggestion @simonatdrg ; LOL its much simpler than my idea of the TokenFilter :-)

from solrtexttagger.

Yes, my field type is little different from the one suggested in the QUICK_START.md. That is why I wasn't sure if you saw the same thing I was seeing. As you pointed out, I guess the positionIncrementGap setting is working for me here.

In addition to the positionIncrementGap I also have omitTermFreqAndPositions set. I copied these off my previous configuration (versions Solr 5.0.0 and SolrTextTagger 2.3-SNAPSHOT). Not sure if these used to be the recommended settings earlier, although I do remember doing some tuning based on my use case at the time.

Also my analysis chain is a little different since I explicitly handle mixed case and lower case (in my application). Also my name_tag analog is multiValued since I explicitly populate it from code (i.e. not a copy-field directive) using all my names (primary + all alternates).

"add-field-type":{
"name" : "tag",
"class" : "solr.TextField",
"positionIncrementGap" : "100",
"postingsFormat" : "Memory",
"omitTermFreqAndPositions" : true,
"omitNorms":true,
"indexAnalyzer":{
"tokenizer":{
"class":"solr.StandardTokenizerFactory" },
"filters":[
{"class":"solr.ASCIIFoldingFilterFactory"},
{"class":"solr.EnglishPossessiveFilterFactory"},
{"class":"org.opensextant.solrtexttagger.ConcatenateFilterFactory"}
]},
"queryAnalyzer":{
"tokenizer":{
"class":"solr.StandardTokenizerFactory" },
"filters":[
{"class":"solr.ASCIIFoldingFilterFactory"},
{"class":"solr.EnglishPossessiveFilterFactory"}
]}
},

from solrtexttagger.