Clean API to expose Lucene Analyzers about pyserini HOT 1 CLOSED

BTW, on the Java end, SimpleSearcher already exposes analyzeWithAnalyzer, so this can be done on the Python end without a new jar:

https://github.com/castorini/anserini/blob/master/src/main/java/io/anserini/index/IndexReaderUtils.java#L222

In fact, it just dispatches to AnalyzerUtils.tokenize(analyzer, text) so the Python end can call that method directly.

I propose an API along the lines of:

analyzer1 = Analyzer()  # default
analyzer1.analyze('this is a string')

analyzer2 = Analyzer('whitespace')  # specify a custom one
analyzer2.analyze('this is a string')

That is, each object is stateful, even though it just dispatches to static methods in Java.

@chriskamphuis can you work on this?

from pyserini.