Status: IN
Deduplication is two-tiered: the Bloom filter rejects exact-URL revisits first (O(k) per check), then SimHash rejects near-duplicate content (linear scan over seen hashes) — content dedup only runs for URLs that pass the Bloom filter.
Source: entries/2026/06/05/web-crawler-web_crawler.md