cluster_pairwise_predictions_at_threshold shows me links below threshold_match_probability

[francisduval]

Hello Splink community!

I have this code:

df_clusters = linker.clustering.cluster_pairwise_predictions_at_threshold(
    df_predictions, threshold_match_probability=0.95
)

linker.visualisations.cluster_studio_dashboard(
    df_predictions,
    df_clusters,
    "cluster_studio.html",
    sampling_method="by_cluster_size",
    overwrite=True,
)

# You can view the scv.html file in your browser, or inline in a notbook as follows
from IPython.display import IFrame

IFrame(src="./cluster_studio.html", width="100%", height=1000)

Since I set threshold_match_probability=0.95, I expect the tool to show me only clusters for which the probability match between nodes is higher than 0.95. However, the tool shows me link below that threshold.

Do you have any idea why?

Thanks!

[ADBond]

When we cluster in Splink we use the 'connected components' algorithm - effectively ignoring any links below the threshold, we cluster together any records that are still connected to each other even if this happens via another record. For example, with an edges table:

record_l	record_r	match_probability
A	B	0.99
A	C	0.8
B	C	0.995

clustering at a threshold of 0.95 we would find all three records in the same cluster:

record	cluster
A	A
B	A
C	A

This is because although A and C are only scored as 0.8, A is linked to B above the threshold, and B is linked to C. So A and C end up in the same cluster through this transitive link. This is a feature; in most cases like this A and C will be a true link that is just not captured by your model for some reason - for instance if there are not enough non-null columns in common to be able to get above the threshold.

By default cluster studio will show you all links that exist in the edges data you supply to it, regardless of whether or not it falls below the clustering threshold. This can be very useful for model QA - if you find many clusters with lots of links below this threshold, it may be a sign of issues with your model (perhaps the high-probability links you have could be false positives, or maybe your low-probability links are false negatives because your model is missing nuance).

If you want to view clusters without these links there is a slider in the controls section that allows you to filter out all links below a given threshold. If you really wanted to remove them entirely from the dashboard you could filter your edges table before passing it in to creat the dashboard.

[francisduval]

Hello ADBond,

Thank you so much for your fast answer! These are crystal clear explanations, makes sense!