{"id":"probabilistic-dedup-trades-memory-for-coverage","text":"The crawler's multi-layer dedup combines deterministic normalization with probabilistic structures (Bloom filter for O(k) exact-URL rejection, SimHash for O(1) near-content detection), achieving coverage across three dimensions while maintaining bounded memory through tunable accuracy-space tradeoffs.","truth_value":"IN","source":"","source_url":"","source_hash":"","justifications":[{"type":"SL","antecedents":["crawler-three-layer-dedup","probabilistic-structures-trade-accuracy-for-space"],"outlist":[],"label":"Each dedup layer uses a different probabilistic tradeoff; bounded memory is the unifying constraint across all three"}],"dependents":["dedup-is-stratified-across-boundaries-and-accuracy-levels","resource-bounding-uses-dual-fidelity-strategies"],"metadata":{},"created_at":"","updated_at":"","reviewed_at":"","verified_at":"","retracted_at":"","explanation":{"steps":[{"node":"probabilistic-dedup-trades-memory-for-coverage","truth_value":"IN","reason":"SL justification valid","antecedents":["crawler-three-layer-dedup","probabilistic-structures-trade-accuracy-for-space"],"label":"Each dedup layer uses a different probabilistic tradeoff; bounded memory is the unifying constraint across all three"},{"node":"crawler-three-layer-dedup","truth_value":"IN","reason":"SL justification valid","antecedents":["crawler-layered-dedup-bloom-then-simhash","bloom-filter-prevents-frontier-duplicates","crawler-normalize-once-convention"],"label":"Each layer catches what the others miss — normalization handles canonicalization, Bloom handles exact revisits, SimHash handles content clones"},{"node":"crawler-layered-dedup-bloom-then-simhash","truth_value":"IN","reason":"premise"},{"node":"bloom-filter-prevents-frontier-duplicates","truth_value":"IN","reason":"premise"},{"node":"crawler-normalize-once-convention","truth_value":"IN","reason":"premise"},{"node":"probabilistic-structures-trade-accuracy-for-space","truth_value":"IN","reason":"SL justification valid","antecedents":["hll-default-precision","morris-counter-32-estimators","simhash-threshold-default-3"],"label":"HLL, Morris, and SimHash all trade tunable accuracy for sub-linear space"},{"node":"hll-default-precision","truth_value":"IN","reason":"premise"},{"node":"morris-counter-32-estimators","truth_value":"IN","reason":"premise"},{"node":"simhash-threshold-default-3","truth_value":"IN","reason":"premise"}]}}