{"id":"dedup-is-stratified-across-boundaries-and-accuracy-levels","text":"The architecture applies dedup at three independent system boundaries with accuracy adapted to cost: exact key-based dedup at API boundaries (idempotency keys for hotel, payment, ad-click), exact event-based dedup at stream processing boundaries (coordinated dedup with watermark finalization), and approximate content-based dedup at crawling boundaries (Bloom filter + SimHash) — each boundary uses the mechanism whose accuracy-memory tradeoff fits its domain's scale and failure cost.","truth_value":"IN","source":"","source_url":"","source_hash":"","justifications":[{"type":"SL","antecedents":["idempotency-keys-ignore-payload-content","forward-only-stream-processing-is-exactly-once","probabilistic-dedup-trades-memory-for-coverage"],"outlist":[],"label":"Three dedup mechanisms at three boundaries — API (exact/key), stream (exact/event), crawl (approximate/content)"}],"dependents":["duplicate-prevention-is-complete-from-api-to-storage"],"metadata":{"last_reviewed":"2026-06-06T06:26:57","review_result":"pass"},"created_at":"","updated_at":"","reviewed_at":"","verified_at":"","retracted_at":"","explanation":{"steps":[{"node":"dedup-is-stratified-across-boundaries-and-accuracy-levels","truth_value":"IN","reason":"SL justification valid","antecedents":["idempotency-keys-ignore-payload-content","forward-only-stream-processing-is-exactly-once","probabilistic-dedup-trades-memory-for-coverage"],"label":"Three dedup mechanisms at three boundaries — API (exact/key), stream (exact/event), crawl (approximate/content)"},{"node":"idempotency-keys-ignore-payload-content","truth_value":"IN","reason":"SL justification valid","antecedents":["hotel-idempotency-ignores-params","payment-idempotency-is-key-based","ad-click-dedup-global-not-per-ad"],"label":"hotel returns cached reservation ignoring guest/dates/room, payment maps key→payment ID without param check, ad-click dedup keys on event_id alone — all three trade payload-awareness for implementation simplicity"},{"node":"hotel-idempotency-ignores-params","truth_value":"IN","reason":"premise"},{"node":"payment-idempotency-is-key-based","truth_value":"IN","reason":"premise"},{"node":"ad-click-dedup-global-not-per-ad","truth_value":"IN","reason":"premise"},{"node":"forward-only-stream-processing-is-exactly-once","truth_value":"IN","reason":"SL justification valid","antecedents":["dedup-and-finalization-are-coordinated","forward-only-design-prevents-regression-and-maximizes-progress"],"label":"Global dedup keying (event_id alone, not per ad_id) means two legitimate events sharing an ID across different entities are silently merged, breaking per-entity exactly-once semantics","outlist":["dedup-is-global-not-per-ad"]},{"node":"dedup-and-finalization-are-coordinated","truth_value":"IN","reason":"SL justification valid","antecedents":["dedup-outlives-aggregation-window","watermark-finalization-is-irreversible"],"label":"Irreversible finalization demands that dedup outlive the window — these two depth-1 conclusions are not independent but structurally dependent"},{"node":"dedup-outlives-aggregation-window","truth_value":"IN","reason":"SL justification valid","antecedents":["dedup-pruning-uses-2x-lateness","watermark-drives-finalization"],"label":"The 2× multiplier ensures dedup coverage extends beyond the point where corrections would be impossible"},{"node":"dedup-pruning-uses-2x-lateness","truth_value":"IN","reason":"premise"},{"node":"watermark-drives-finalization","truth_value":"IN","reason":"premise"},{"node":"watermark-finalization-is-irreversible","truth_value":"IN","reason":"SL justification valid","antecedents":["watermark-drives-finalization","window-lifecycle-one-directional","no-window-merging-or-retraction"],"label":"Three properties compose into hard irreversibility: watermark-only trigger + one-way lifecycle + no retraction"},{"node":"window-lifecycle-one-directional","truth_value":"IN","reason":"premise"},{"node":"no-window-merging-or-retraction","truth_value":"IN","reason":"premise"},{"node":"forward-only-design-prevents-regression-and-maximizes-progress","truth_value":"IN","reason":"SL justification valid","antecedents":["state-ratchets-prevent-regression-across-domains","pipeline-processing-maximizes-forward-progress"],"label":"State ratchets and forward-progress pipelines are complementary mechanisms — ratchets prevent regression in stateful systems, pipelines prevent regression in dataflow systems"},{"node":"state-ratchets-prevent-regression-across-domains","truth_value":"IN","reason":"SL justification valid","antecedents":["chat-monotonic-read-progress","watermark-finalization-is-irreversible"],"label":"chat read cursors are monotonic (never re-mark as unread), aggregation windows follow OPEN→CLOSED→FINALIZED with no reversal — both are state ratchets that make backwards movement structurally impossible"},{"node":"chat-monotonic-read-progress","truth_value":"IN","reason":"SL justification valid","antecedents":["chat-dual-ordering-sequence-and-lamport","chat-read-cursors-monotonic"],"label":"Sequence-number ordering + monotonic cursors create an irreversible read-progress guarantee"},{"node":"chat-dual-ordering-sequence-and-lamport","truth_value":"IN","reason":"premise"},{"node":"chat-read-cursors-monotonic","truth_value":"IN","reason":"premise"},{"node":"pipeline-processing-maximizes-forward-progress","truth_value":"IN","reason":"SL justification valid","antecedents":["video-pipeline-maximizes-useful-work-on-failure","watermark-finalization-is-irreversible"],"label":"Video pipeline (branch-independent failure containment) and stream processing (irreversible finalization) both embody the same forward-only processing principle"},{"node":"video-pipeline-maximizes-useful-work-on-failure","truth_value":"IN","reason":"SL justification valid","antecedents":["dag-failure-cascade","youtube-pipeline-dag-structure"],"label":"DAG topology + selective cascade means a thumbnail failure doesn't block a successful transcode"},{"node":"dag-failure-cascade","truth_value":"IN","reason":"premise"},{"node":"youtube-pipeline-dag-structure","truth_value":"IN","reason":"premise"},{"node":"probabilistic-dedup-trades-memory-for-coverage","truth_value":"IN","reason":"SL justification valid","antecedents":["crawler-three-layer-dedup","probabilistic-structures-trade-accuracy-for-space"],"label":"Each dedup layer uses a different probabilistic tradeoff; bounded memory is the unifying constraint across all three"},{"node":"crawler-three-layer-dedup","truth_value":"IN","reason":"SL justification valid","antecedents":["crawler-layered-dedup-bloom-then-simhash","bloom-filter-prevents-frontier-duplicates","crawler-normalize-once-convention"],"label":"Each layer catches what the others miss — normalization handles canonicalization, Bloom handles exact revisits, SimHash handles content clones"},{"node":"crawler-layered-dedup-bloom-then-simhash","truth_value":"IN","reason":"premise"},{"node":"bloom-filter-prevents-frontier-duplicates","truth_value":"IN","reason":"premise"},{"node":"crawler-normalize-once-convention","truth_value":"IN","reason":"premise"},{"node":"probabilistic-structures-trade-accuracy-for-space","truth_value":"IN","reason":"SL justification valid","antecedents":["hll-default-precision","morris-counter-32-estimators","simhash-threshold-default-3"],"label":"HLL, Morris, and SimHash all trade tunable accuracy for sub-linear space"},{"node":"hll-default-precision","truth_value":"IN","reason":"premise"},{"node":"morris-counter-32-estimators","truth_value":"IN","reason":"premise"},{"node":"simhash-threshold-default-3","truth_value":"IN","reason":"premise"}]}}