understanding similitude differences between extract, cdist and extractOne about rapidfuzz HOT 4 CLOSED

for sco in ['fuzz.ratio','fuzz.WRatio','fuzz.QRatio']:
    result = extract(query = x, choices = choices, score_cutoff=90, scorer=eval(sco))

why would you use eval for this 😨 You can simply iterate over the functions:

for scorer in [fuzz.ratio, fuzz.WRatio, fuzz.QRatio]:
    result = extract(x, choices, score_cutoff=90, scorer=scorer)

Why does extract() return an empty list from some of the explicit scorer values I tried? (fuzz.ratio and fuzz.QRatio)

process.extract filters out results if you pass a score_cutoff. In your case fuzz.ratio/fuzz.QRatio return a similarity below 90 for each element.

Which scorer should I use in first test, so similitudes are not the same? (can't understand why all matches get a 90.0 score)

A lot of the scorers use a partial match and since:

• 'Soliva sessilis' is a substring of 'Soliva sessilis auct., non Ruiz & Pav.'
• 'Soliva sessilis' is a substring of 'Soliva sessilis Ruiz & Pav.'
• 'Soliva' is a substring of 'Soliva sessilis'

This partial match always has a similarity of 100. fuzz.WRatio puts a weight of 0.9 on these partial matches, so you end up with a similarity of 90. E.g. fuzz.ratio or the distances in rapidfuzz.distance.* do not show this behaviour.

Why 2nd and 3rd example produce a 71.42857 similitude of 'Soliva sessilis' against 'Soliva sessilis Ruiz & Pav.', whereas in 1st example the output is 90, as I said above?

fuzz.ratio and fuzz.QRatio are based on the normalized Indel similarity. Not sure whether you have any specific questions into how they work.

from rapidfuzz.

Thanks a lot for your prompt answer and explanations @maxbachmann !

why would you use eval for this 😨 You can simply iterate over the functions

I knew, but I didn't know how to output functions' names in the next line of code.
I was rushy to post the question so I used strings for the names, and eval() for this simple test.
Now I know I should use function.__name__.

In your case fuzz.ratio/fuzz.QRatio return a similarity below 90 for each element.

Thanks, So I need to lower my cutoff and also read more carefully the defaults.
I misread and thought fuzz.ratio was the default scorer value for extract() so the 1st two lines of my test 1 output looked inconsistent to me (but its default is actually fuzz.WRatio, which explains their different output).
Changed cutoff to 70 and this is the output of 1st test:

String list (choices):  ['Soliva sessilis auct., non Ruiz & Pav.', 'Soliva sessilis Ruiz & Pav.', 'Soliva']

- Result with no scorer (default scorer):
 [('Soliva sessilis auct., non Ruiz & Pav.', 90.0, 0), ('Soliva sessilis Ruiz & Pav.', 90.0, 1), ('Soliva', 90.0, 2)]

+ Result when explicitly passing scorer=fuzz.ratio:
 [('Soliva sessilis Ruiz & Pav.', 71.42857142857143, 1)]

+ Result when explicitly passing scorer=fuzz.WRatio:
 [('Soliva sessilis auct., non Ruiz & Pav.', 90.0, 0), ('Soliva sessilis Ruiz & Pav.', 90.0, 1), ('Soliva', 90.0, 2)]

+ Result when explicitly passing scorer=fuzz.QRatio:
 [('Soliva sessilis Ruiz & Pav.', 71.42857142857143, 1)]

That makes clear that I should use either choose fuzz.ratio or fuzz.QRatio for my use case.

fuzz.ratio and fuzz.QRatio are based on the normalized Indel similarity. Not sure whether you have any specific questions into how they work.

If it's not too advanced stuff yes, I'd like to know their difference (when using one or the other).
But I guess I'll better get how it works by example.

With my sample data above they both produce the same similarity values.
Any particular modification of my query that would produce differences between fuzz.ratio and fuzz.QRatio outputs?
... so I can figure out which of them suits better to what I want to do.

EDIT:
I had also pasted my example data but I guess this is not related to the issue title.
I better move it to discussions #347

from rapidfuzz.

I misread and thought fuzz.ratio was the default scorer value for extract() so the 1st two lines of my test 1 output looked inconsistent to me (but its default is actually fuzz.WRatio, which explains their different output).

Yes this is a bit inconsistent, since cdist defaults to fuzz.ratio, while extract and extractOne default to fuzz.WRatio.

Any particular modification of my query that would produce differences between fuzz.ratio and fuzz.QRatio outputs?

They are pretty much exactly the same. The only difference between the two is the handling of empty strings.

>>> from rapidfuzz import fuzz
>>> fuzz.ratio("", "")
100.0
>>> fuzz.QRatio("", "")
0.0

In the past they had different defaults for the preprocessing function, but this was changed in v3.0.0.

from rapidfuzz.

Thanks a lot for explaining.
Issue solved!

from rapidfuzz.