materialization-barriers-span-storage-and-processing

Status: IN

Unbounded in-memory materialization is a cross-cutting concern from storage through batch processing: storage operations (LSM range scans, compaction, SSTable index rebuilds) materialize entire datasets, and pipeline stages (Count) silently accumulate all input before producing output, meaning the materialization problem exists at every layer of the data processing stack.

Justifications

Both storage reads and batch processing stages have unbounded materialization points

Depends on (SL): storage-operations-have-unbounded-memory-consumption, count-is-barrier

Depended on by

JSON