Support for position metadata in Japanese about lunr-languages HOT 2 CLOSED

I think the problem is that the Japanese language plugin must provide its own tokenizer, due to the extra work required for correctly tokenising Japanese text. This means it isn't recording the same metadata that the built in tokeniser does.

Here is how it is done in Lunr - https://github.com/olivernn/lunr.js/blob/master/lib/tokenizer.js#L41-L46
And here is the tokeniser in the Japanese adaptor - https://github.com/MihaiValentin/lunr-languages/blob/master/lunr.ja.js#L105

So you would have to understand how to get positions out of tool being used to segment the Japanese text, I'm not familiar with that library so unable to say how easy that would be, but adding them to the index is just a matter of passing the details to the lunr.Token constructor

from lunr-languages.

Hey @olivernn!

I was just looking through the JA plugin for lunr-languages and after comparing it with the base tokenizer code for EN it seems pretty straightforward to attach the position metadata for Japanese. I'm going to try and work through it tomorrow and I'll update this issue accordingly.

Thank you for all your work on lunr!

from lunr-languages.