Status: IN
The crawler's multi-layer dedup combines deterministic normalization with probabilistic structures (Bloom filter for O(k) exact-URL rejection, SimHash for O(1) near-content detection), achieving coverage across three dimensions while maintaining bounded memory through tunable accuracy-space tradeoffs.
Each dedup layer uses a different probabilistic tradeoff; bounded memory is the unifying constraint across all three
Depends on (SL): crawler-three-layer-dedup, probabilistic-structures-trade-accuracy-for-space