Reimplement FTS using an external library (SQLite? CLucene?) about cbforest HOT 7 OPEN

Here are some performance comparisons between SQLite, a couple of flavors of Lucene, and one I hadn't heard of called Tracker. SQLite and CLucene are roughly equivalent, and since SQLite's effective code size is nearly zero, it seems like the best choice.

from cbforest.

SQLite's FTS comes with a pretty basic tokenizer that doesn't handle stop-words or stemming. We've been using the fts3-unicodesn library for that, which incorporates Snowball stemmers for a dozen languages, plus a custom list of stop-words.

It would be good to make this pluggable, so we can use OS facilities instead. For example, the Cocoa Foundation framework has classes for doing tokenization and stemming, which I believe support many more languages, including Asian ones which are very difficult to work with.

from cbforest.

There's also Xapian, which has a nice feature set.

from cbforest.

the Cocoa Foundation framework has classes for doing tokenization and stemming, which I believe support many more languages, including Asian ones which are very difficult to work with.

This would be amazing to have as I have a lot of Asian customers.

from cbforest.

I'm very interested in this feature, since I'm using couchbase with ForestDB and currently word-prefix matching is not supported. Is there any time frame for implementing this feature?

from cbforest.

No, it's not on our roadmap so far. Of course I'd be happy to advise & assist someone else who wanted to work on it…

from cbforest.

@lucatorella Word-prefix matching is the only reason why I haven't switched to ForestDB just yet.

from cbforest.