Term frequency adjustment calculation is flawed about splink HOT 4 CLOSED

The plan is to implement ☝️ as a fix, with some functional tests as mentioned above

from splink.

With all this in mind, term freqnecy adjustment can be expressed as a simple extension to the standard formula for computing m, u probabilities and adjustment factors, as follows:

from splink.

The existing approach to term frequency adjustments replaces the base lambda (the proportion of matches across all records) with the proportion of matches within a name, which is our interpretation of the final formula on page 9 of https://static.cambridge.org/content/id/urn:cambridge.org:id:article:S0003055418000783/resource/name/S0003055418000783sup001.pdf.

However, I am worried that this double counts the adjustment for gamma_name = 1 i.e. we shouldn't adjust up for the fact that name matches, and ALSO adjust up for the prevalance of matches within a specic name. Sadly there's a typo (missing term) in the formula above, so it's not clear exactly how it should be performed.

There's a missing term that could be something like:

Except perhaps it's meant to be 'the product over all k, except for the first_name k.

from splink.

Here are some properties we agree should be true of the term frequency adjustments:

• 'term frequency adjustments' made because of skew in the data. the match_probability already takes account of cardinality in the data. The problem is they assume that each value appears roughly an equal amount of times.

• e.g. we know that a national insurance number contributes a huge amount more informaion to a match than gender because of its high cardinality. What about if a column had half binary values, and half NI numbers?

• this implies that when term frequency adjustments are made, match probability should always be expected to be adjusted up for some values of name and down for others. This is somethign we can run as a unit test.

• for a column without skew, we expect term frequency adjustments to be 'null'. Aggain, we can test this

from splink.