dcjanus / cang-jie Goto Github PK

ref:

• quickwit-oss/tantivy#483
• quickwit-oss/tantivy@6a547b0#diff-80398c5faae3c069e4e6aa2ed11b28c0R43
  solution: work with tantivy 0.9

但源码是升级了依赖再bump version的

The problem is that when I want to create a snippet of tantivy query result, an error(byte index xx is out of bounds of xxxx) will be raised. Looks like an out-of-bounds access to the string(byte array).

This is my test code: https://gist.github.com/WincerChan/09a1777944914164f6a22b62c95ec564

cargo dependencies:

1. tantivy = "0.14.0"
2. cang-jie = "0.11.0"
3. jieba-rs = { version = "0.6.2", default-features = false }

After some works, I find the problem is token stream: https://github.com/DCjanus/cang-jie/blob/master/src/stream.rs#L30

The variable offset_from does not start from 0, but from current_word.len(), So there will be an out-of-bounds error at the last token.

If necessary, I can make a PR to fix it.

tantivy发布了0.16版本，根据其在CHANGELOG中的描述，仅仅是bugfix，所以可以安全地bump version。