storage-operations-have-unbounded-memory-consumption

Status: IN

Three core storage operations materialize entire datasets in memory with no streaming, pagination, or backpressure: LSM range scans load all matching entries into a dict before returning, compaction buffers all merged entries into a list before writing, and SSTable lookups rebuild the sparse index key list on every call — creating O(n) memory pressure proportional to total data size rather than result size.

Justifications

All three operations eagerly materialize full datasets with no memory bound

Depends on (SL): lsm-range-scan-materializes-all, compaction-buffers-all-entries, sstable-get-rebuilds-key-list-every-call

Depended on by

JSON