D-Swoosh: A Family of Algorithms for Generic, Distributed Entity Resolution

Full textClick to download.
CitationStanford University Technical Report
AuthorsOmar Benjelloun
Hector Garcia-Molina
Hideki Kawai
Tait Larson
David Menestrina
Sutthipong Thavisomboon


Entity Resolution (ER) matches and merges records that refer to the same real-world entities, and is typically a compute-intensive process due to complex matching functions and high data volumes. We present a family of algorithms, D-Swoosh, for distributing the ER workload across multiple processors. The algorithms use generic match and merge functions, and ensure that new merged records are distributed to processors that may have matching records. We perform a detailed performance evaluation, for cases where application knowledge can eliminate some comparisons, and for cases where all records must be matched. Our experiments use actual comparison shopping data provided by Yahoo!.

Back to publications
Back to previous page