{"id":"crawler-three-layer-dedup","text":"The web crawler achieves comprehensive deduplication through three coordinated layers: URL normalization at ingestion boundaries, Bloom filter for O(k) exact-URL rejection, and SimHash for near-duplicate content detection.","truth_value":"IN","source":"","source_url":"","source_hash":"","justifications":[{"type":"SL","antecedents":["crawler-layered-dedup-bloom-then-simhash","bloom-filter-prevents-frontier-duplicates","crawler-normalize-once-convention"],"outlist":[],"label":"Each layer catches what the others miss — normalization handles canonicalization, Bloom handles exact revisits, SimHash handles content clones"}],"dependents":["probabilistic-dedup-trades-memory-for-coverage"],"metadata":{"last_reviewed":"2026-06-06T06:26:57","review_result":"unnecessary"},"created_at":"","updated_at":"","reviewed_at":"","verified_at":"","retracted_at":"","explanation":{"steps":[{"node":"crawler-three-layer-dedup","truth_value":"IN","reason":"SL justification valid","antecedents":["crawler-layered-dedup-bloom-then-simhash","bloom-filter-prevents-frontier-duplicates","crawler-normalize-once-convention"],"label":"Each layer catches what the others miss — normalization handles canonicalization, Bloom handles exact revisits, SimHash handles content clones"},{"node":"crawler-layered-dedup-bloom-then-simhash","truth_value":"IN","reason":"premise"},{"node":"bloom-filter-prevents-frontier-duplicates","truth_value":"IN","reason":"premise"},{"node":"crawler-normalize-once-convention","truth_value":"IN","reason":"premise"}]}}