{"id":"crawler-layered-dedup-bloom-then-simhash","text":"Deduplication is two-tiered: the Bloom filter rejects exact-URL revisits first (O(k) per check), then SimHash rejects near-duplicate content (linear scan over seen hashes) — content dedup only runs for URLs that pass the Bloom filter.","truth_value":"IN","source":"entries/2026/06/05/web-crawler-web_crawler.md","source_url":"","source_hash":"","justifications":[],"dependents":["crawler-three-layer-dedup"],"metadata":{},"created_at":"","updated_at":"","reviewed_at":"","verified_at":"","retracted_at":"","explanation":{"steps":[{"node":"crawler-layered-dedup-bloom-then-simhash","truth_value":"IN","reason":"premise"}]}}