MinJoin: Efficient Edit Similarity Joins via Local Hash Minimums
share
In this paper we study edit similarity joins, in which we are given a set of strings and a threshold parameter K, and asked to find all pairs of strings whose edit distance are at most K. Edit similarity joins is a fundamental problem in query processing, and has been studied extensively in the database community since early 2000s. However, all previous algorithms either cannot scale well to long strings and large distance thresholds, or suffer from imperfect accuracy. In this paper we propose a novel randomized algorithm based on string partitions using local minimum hash values. We provide a thorough theoretical analysis for our algorithm and an extensive set of experiments on real world datasets. We found that our algorithm significantly outperforms all existing deterministic algorithms on long strings and large distance thresholds, while achieves perfect accuracy on all the datasets that we have tested.
READ FULL TEXT