{"id":"duplicate-prevention-is-complete-from-api-to-storage","text":"The architecture achieves complete duplicate prevention from external API boundary to internal storage through complementary forward-only mechanisms: idempotency keys extend forward-only semantics to the client boundary (making duplicate submissions return cached results permanently), while stratified dedup covers internal processing with accuracy adapted to cost (exact key-based at API boundaries, coordinated window-based at stream boundaries, probabilistic at crawl boundaries).","truth_value":"IN","source":"","source_url":"","source_hash":"","justifications":[{"type":"SL","antecedents":["idempotency-extends-forward-only-to-client-boundary","dedup-is-stratified-across-boundaries-and-accuracy-levels"],"outlist":[],"label":"Client-facing idempotency and internal stratified dedup together cover the full duplicate prevention surface"}],"dependents":[],"metadata":{"last_reviewed":"2026-06-06T06:26:57","review_result":"pass"},"created_at":"","updated_at":"","reviewed_at":"","verified_at":"","retracted_at":"","explanation":{"steps":[{"node":"duplicate-prevention-is-complete-from-api-to-storage","truth_value":"IN","reason":"SL justification valid","antecedents":["idempotency-extends-forward-only-to-client-boundary","dedup-is-stratified-across-boundaries-and-accuracy-levels"],"label":"Client-facing idempotency and internal stratified dedup together cover the full duplicate prevention surface"},{"node":"idempotency-extends-forward-only-to-client-boundary","truth_value":"IN","reason":"SL justification valid","antecedents":["idempotency-keys-ignore-payload-content","forward-only-extends-to-failure-handling"],"label":"Idempotency is forward-only applied to external client retries"},{"node":"idempotency-keys-ignore-payload-content","truth_value":"IN","reason":"SL justification valid","antecedents":["hotel-idempotency-ignores-params","payment-idempotency-is-key-based","ad-click-dedup-global-not-per-ad"],"label":"hotel returns cached reservation ignoring guest/dates/room, payment maps key→payment ID without param check, ad-click dedup keys on event_id alone — all three trade payload-awareness for implementation simplicity"},{"node":"hotel-idempotency-ignores-params","truth_value":"IN","reason":"premise"},{"node":"payment-idempotency-is-key-based","truth_value":"IN","reason":"premise"},{"node":"ad-click-dedup-global-not-per-ad","truth_value":"IN","reason":"premise"},{"node":"forward-only-extends-to-failure-handling","truth_value":"IN","reason":"SL justification valid","antecedents":["forward-only-design-prevents-regression-and-maximizes-progress","retry-escalates-to-permanent-failure"],"label":"The d3 forward-only node covers normal state progression; the d1 retry node covers failure progression — the emergent property is that both follow the same no-rollback principle"},{"node":"forward-only-design-prevents-regression-and-maximizes-progress","truth_value":"IN","reason":"SL justification valid","antecedents":["state-ratchets-prevent-regression-across-domains","pipeline-processing-maximizes-forward-progress"],"label":"State ratchets and forward-progress pipelines are complementary mechanisms — ratchets prevent regression in stateful systems, pipelines prevent regression in dataflow systems"},{"node":"state-ratchets-prevent-regression-across-domains","truth_value":"IN","reason":"SL justification valid","antecedents":["chat-monotonic-read-progress","watermark-finalization-is-irreversible"],"label":"chat read cursors are monotonic (never re-mark as unread), aggregation windows follow OPEN→CLOSED→FINALIZED with no reversal — both are state ratchets that make backwards movement structurally impossible"},{"node":"chat-monotonic-read-progress","truth_value":"IN","reason":"SL justification valid","antecedents":["chat-dual-ordering-sequence-and-lamport","chat-read-cursors-monotonic"],"label":"Sequence-number ordering + monotonic cursors create an irreversible read-progress guarantee"},{"node":"chat-dual-ordering-sequence-and-lamport","truth_value":"IN","reason":"premise"},{"node":"chat-read-cursors-monotonic","truth_value":"IN","reason":"premise"},{"node":"watermark-finalization-is-irreversible","truth_value":"IN","reason":"SL justification valid","antecedents":["watermark-drives-finalization","window-lifecycle-one-directional","no-window-merging-or-retraction"],"label":"Three properties compose into hard irreversibility: watermark-only trigger + one-way lifecycle + no retraction"},{"node":"watermark-drives-finalization","truth_value":"IN","reason":"premise"},{"node":"window-lifecycle-one-directional","truth_value":"IN","reason":"premise"},{"node":"no-window-merging-or-retraction","truth_value":"IN","reason":"premise"},{"node":"pipeline-processing-maximizes-forward-progress","truth_value":"IN","reason":"SL justification valid","antecedents":["video-pipeline-maximizes-useful-work-on-failure","watermark-finalization-is-irreversible"],"label":"Video pipeline (branch-independent failure containment) and stream processing (irreversible finalization) both embody the same forward-only processing principle"},{"node":"video-pipeline-maximizes-useful-work-on-failure","truth_value":"IN","reason":"SL justification valid","antecedents":["dag-failure-cascade","youtube-pipeline-dag-structure"],"label":"DAG topology + selective cascade means a thumbnail failure doesn't block a successful transcode"},{"node":"dag-failure-cascade","truth_value":"IN","reason":"premise"},{"node":"youtube-pipeline-dag-structure","truth_value":"IN","reason":"premise"},{"node":"retry-escalates-to-permanent-failure","truth_value":"IN","reason":"SL justification valid","antecedents":["notif-exponential-backoff-with-jitter","retry-converts-timeout-to-failure"],"label":"Backoff prevents cascading load; finite retry count ensures eventual resolution (success or permanent failure) rather than indefinite limbo"},{"node":"notif-exponential-backoff-with-jitter","truth_value":"IN","reason":"premise"},{"node":"retry-converts-timeout-to-failure","truth_value":"IN","reason":"premise"},{"node":"dedup-is-stratified-across-boundaries-and-accuracy-levels","truth_value":"IN","reason":"SL justification valid","antecedents":["idempotency-keys-ignore-payload-content","forward-only-stream-processing-is-exactly-once","probabilistic-dedup-trades-memory-for-coverage"],"label":"Three dedup mechanisms at three boundaries — API (exact/key), stream (exact/event), crawl (approximate/content)"},{"node":"forward-only-stream-processing-is-exactly-once","truth_value":"IN","reason":"SL justification valid","antecedents":["dedup-and-finalization-are-coordinated","forward-only-design-prevents-regression-and-maximizes-progress"],"label":"Global dedup keying (event_id alone, not per ad_id) means two legitimate events sharing an ID across different entities are silently merged, breaking per-entity exactly-once semantics","outlist":["dedup-is-global-not-per-ad"]},{"node":"dedup-and-finalization-are-coordinated","truth_value":"IN","reason":"SL justification valid","antecedents":["dedup-outlives-aggregation-window","watermark-finalization-is-irreversible"],"label":"Irreversible finalization demands that dedup outlive the window — these two depth-1 conclusions are not independent but structurally dependent"},{"node":"dedup-outlives-aggregation-window","truth_value":"IN","reason":"SL justification valid","antecedents":["dedup-pruning-uses-2x-lateness","watermark-drives-finalization"],"label":"The 2× multiplier ensures dedup coverage extends beyond the point where corrections would be impossible"},{"node":"dedup-pruning-uses-2x-lateness","truth_value":"IN","reason":"premise"},{"node":"probabilistic-dedup-trades-memory-for-coverage","truth_value":"IN","reason":"SL justification valid","antecedents":["crawler-three-layer-dedup","probabilistic-structures-trade-accuracy-for-space"],"label":"Each dedup layer uses a different probabilistic tradeoff; bounded memory is the unifying constraint across all three"},{"node":"crawler-three-layer-dedup","truth_value":"IN","reason":"SL justification valid","antecedents":["crawler-layered-dedup-bloom-then-simhash","bloom-filter-prevents-frontier-duplicates","crawler-normalize-once-convention"],"label":"Each layer catches what the others miss — normalization handles canonicalization, Bloom handles exact revisits, SimHash handles content clones"},{"node":"crawler-layered-dedup-bloom-then-simhash","truth_value":"IN","reason":"premise"},{"node":"bloom-filter-prevents-frontier-duplicates","truth_value":"IN","reason":"premise"},{"node":"crawler-normalize-once-convention","truth_value":"IN","reason":"premise"},{"node":"probabilistic-structures-trade-accuracy-for-space","truth_value":"IN","reason":"SL justification valid","antecedents":["hll-default-precision","morris-counter-32-estimators","simhash-threshold-default-3"],"label":"HLL, Morris, and SimHash all trade tunable accuracy for sub-linear space"},{"node":"hll-default-precision","truth_value":"IN","reason":"premise"},{"node":"morris-counter-32-estimators","truth_value":"IN","reason":"premise"},{"node":"simhash-threshold-default-3","truth_value":"IN","reason":"premise"}]}}