https://blog.nelhage.com/post/fuzzy-dedup/Finding near-duplicates with Jaccard similarity and MinHashSuppose we have a large collection of documents, and we wish you identify which documents are approximately the same as each other. For instance, we may have crawled the web over some period of time, and expect to have fetched the “same page” several times, but to see slight differences in metadata, or that we have several revisions of a page following small edits.
In this post I want to explore the method of approximate deduplication via Jaccard similarity and the MinHash approximation trick.2024-07-03T23:00:00Zhttps://blog.nelhage.com/post/Nelson ElhageRecent content in Posts on Made of BugsPosts on Made of Bugs2026-09-06T20:37:03Zrss20false60Nelson Elhagenelson-elhage0"5b2ae9eec2e293572f6ae5a000bd0161-ssl-df"200utf-8