Name Matching on Large Datasets in Databricks

[ojasch3]

Hi,
I recently came across Zing, it looks like a fantastic product. I see that it has databricks integration. Basically, what I am trying to do is do large name matching between external datasources from which I have ingested into databricks to internal names we have in our own tables. How to go about this? For instance, one use case is I have brought into about 4,000 names and I want to see which of our 6500 names it produces best matches to. Additionally, I want to know how well this would scale because another use case compares 85,000 external names to 300,000 internal names to see what best matches.

[sania-16]

When two datasets are duplicate free and need to be matched against each other, the link phase can be used
For more details - https://docs.zingg.ai/latest/stepbystep/link

You need to set up your input data as both your internal and external source for the link phase - for reference https://github.com/zinggAI/zingg/blob/main/examples/febrl/configLink.json