Can't train for M values on Databricks about splink HOT 4 CLOSED

Wed need to see your full script to be able to help with this, there seems to be something weird going on with �[4m)�[0m

from splink.

Yeah, noticed that too, and let me add it here.

`%pip install 'splink[spark]'
dbutils.library.restartPython()

spark.conf.set("spark.default.parelleism","50")
spark = SparkSession.builder.getOrCreate()

df = spark.sql("SELECT unique_id, column1, creation_date FROM table")

settings = {
"link_type": "dedupe_only",
"comparisons": [
ctl.name_comparison("column1"),
],
"blocking_rules_to_generate_predictions": [
{'blocking_rule':'l.column1= r.column1','salting_partitions':4}
],
"retain_matching_columns": True,
"retain_intermediate_calculation_columns": True,
"sql_dialect":"databricks"
}

linker = SparkLinker(df,settings,spark=spark)
linker.estimate_u_using_random_sampling(max_pairs=1e7)
block_rule = "l.column1= r.column1"
linker.estimate_parameters_using_expectation_maximisation(block_rule)
`

Here's the script so far.

from splink.

You need more than a single comparison, see the note here:
https://github.com/moj-analytical-services/splink?tab=readme-ov-file#what-data-does-splink-work-best-with

from splink.

I got a better understanding now around the comparisons, thanks for clarifying.

from splink.