Make methods in IndexReaderUtils more consistent re: Analyzer about pyserini HOT 10 CLOSED

you can just do:

get_term_counts(self, term: str, analyzer=get_lucene_analyzer()) -> Tuple[int, int]: 
    if analyzer is None:
        # skip analysis (pass dummy to Anserini)
    else:
       # perform analysis with analyzer (either default or custom)
    ...

where you import that function from pyanalysis

from pyserini.

Both taking an analyzer makes sense to me, especially since the getPostingsListWithAnalyzer method is already available in Anserini.

from pyserini.

@PepijnBoers do you have cycles to take this on?

from pyserini.

@lintool sure!

from pyserini.

Currently get_term_counts applies Anserini's default Lucene analyzer if no analyzer is specified (analyzer=None), then in order to skip term analysis you have to pass a dummy analyzer. Is this the requested behavior or do we want analyzer=None to mean that no analysis should take place?

from pyserini.

I'm thinking:

• No analyzer specified - use default.
• analyzer=foo - use foo as analyzer
• analyzer=None - skip analysis

And we make this behavior across all methods (existing, and in the future).

Thoughts?

from pyserini.

I assumed that None would be the value for an unspecified analyzer, do you suggest we overload the methods in Pyserini and remove a default value for the analyzer parameter?

from pyserini.

We could do it like this:

get_term_counts(self, term: str, analyzer='unspecified') -> Tuple[int, int]: 
    if analyzer is 'unspecified'":
        # use default
    elif analyzer is None:
        # skip analysis (pass dummy to Anserini)
    else:
       # perform analysis with given analyzer
    ...

from pyserini.

What about something simpler, like this?

get_term_counts(self, term: str, analyzer=default) -> Tuple[int, int]: 
    if analyzer is None:
        # skip analysis (pass dummy to Anserini)
    else:
       # perform analysis with analyzer (either default or custom)
    ...

from pyserini.

Looks good, but then we have to specify default somewhere, otherwise we face a NameError. The question would then also be how/where to define default, right?

from pyserini.