get_largest_blocks() limited to simple blocking rules about splink HOT 2 CLOSED

Some notes:

abs(l.height - r.height) < 5 is a valid blocking rule, but it’s not one which get_largest_blocks should do analysis of

that’s because it doesn’t actually restrict the comparison space. In order to execute it, spark needs to create a block of records with all combinations of height, and then evaluate the condition for each row in the block

i.e. the block size that we care about is actually the block without that condition

Maybe the function should be less magical, and just be something like get_largest_block_size(columns)

and rather than pass it a blocking rule, you just pass it e.g. get_largest_block_size(["height", "dob"])

none of this means that it’s something you shouldn’t use in your blocking . It does mean, however, that the OTHER conditions associated with that rule should be fairly strict - i.e. you shouldn’t rely on it to restrict the comparison space…or at least, it’s a potential source of slowdown

think of it more like a filter that is applied after the blocks have been computed, rather than a blocking rule

from splink.

Closed by #278

from splink.