Date: 2026-06-05
Time: 13:52
metrics-monitoring-and-alerting/metrics.pyThis file implements an in-memory time-series metrics monitoring and alerting system — the kind of thing you'd build if asked "Design a metrics monitoring system like Datadog or Prometheus" in a system design interview. It owns the entire pipeline: ingestion of metric data points, time-windowed querying with aggregation, alert rule evaluation with state machines, and data lifecycle management (downsampling + retention).
DataPoint — The ingestion primitive. A metric name, float value, timestamp, and optional tags dict. Tags enable multi-dimensional filtering (e.g., {"host": "web-01", "region": "us-east"}).AlertRule — Defines a threshold-based alerting condition. Supports comparisons (gt, lt, gte, lte) and ratechange (percentage change over a window). durationseconds controls how long a condition must hold before firing.Alert — A snapshot of an alert state transition, emitted when evaluate_alerts detects a change.MetricsServiceThe core class. Single-node, in-memory — no persistence or distribution, consistent with the SDI interview scope of demonstrating architecture rather than production readiness.
Storage model: data is a dict keyed by (metricname, frozenset(tags.items())). Values are sorted lists of (timestamp, value) tuples. The frozenset key means each unique tag combination gets its own series — this is the standard time-series cardinality model (identical to how Prometheus stores series by label set).
Ingestion:
ingest() uses bisect.bisect_right to insert in sorted order — O(log n) search but O(n) insertion due to list shifting. Adequate for interview demonstration; a production system would use an append-only buffer with periodic sorting.ingest_batch() is a simple loop over ingest() — no bulk optimization.Querying:
query() supports time-range slicing, bucketed aggregation (sum, avg, min, max, count, percentiles), and group_by for tag-based grouping._slice() uses bisect for O(log n) range lookups on sorted series._aggregate() handles percentile computation via linear interpolation between floor/ceil indices — the standard interpolation method._bucketize() partitions points into fixed-width time buckets.Alerting:
addalertrule() / removealertrule() manage the rule registry.evaluate_alerts() is the poll-based evaluation loop. It checks every rule, drives the state machine, and returns a list of state transitions.on_alert() registers callbacks invoked on state changes.Data Lifecycle:
downsample() compacts old data: 1–7 days old → 5-minute buckets, >7 days old → 1-hour buckets.applyretention() drops data older than retentionseconds (default 30 days). Cleans up empty series from _data.Sorted-list time series with bisect. Every series is kept sorted by timestamp, enabling O(log n) range queries via bisectleft/bisectright. This is the in-memory analog of a time-series database's sorted index. The tuple comparison (timestamp,) works because Python compares tuples element-by-element.
Tag-based series multiplexing. Using frozenset(tags.items()) as part of the key gives each unique label set its own series, then matchingkeys does a linear scan with subset matching for queries. This mirrors how Prometheus/Datadog handle label cardinality — each unique label combination is a distinct time series.
Alert state machine. Four states: OK → PENDING → ALERTING → RESOLVED → (back to OK or PENDING). The PENDING state implements the duration_seconds "for" clause — the condition must persist for the specified duration before firing. This prevents noisy alerts from transient spikes.
Observer pattern for alert callbacks. callbacks list with onalert() registration. Simple pub-sub for notification delivery.
Imports: Only stdlib — bisect for sorted-list operations, math for percentile floor/ceil, dataclasses for data modeling. No external dependencies.
Imported by: testmetrics.py and testsmoke.py — the test suites.
DataPoint → ingest() → compute (metric, frozenset(tags)) key → bisect.bisect_right to find insert position → list.insert in sorted order.
query() → matchingkeys() scans data for matching metric + tag subset → slice() each matching series → merge + sort across series → bucketize() into time windows → aggregate() each bucket → return list of {timestamp, value} dicts.
evaluatealerts(currenttime) → for each rule: checkcondition() gathers data in the lookback window → computes avg (or rate for rate_change) → compare to threshold → drive state machine → emit Alert objects for transitions → invoke callbacks.
downsample(current_time) → for each series, apply age-based bucketing rules → replace raw points with averaged buckets.
applyretention(currenttime) → bisect to find cutoff index → truncate series → delete empty keys.
1. Series are always sorted by timestamp. ingest() maintains this via bisectright insertion. downsample() and applyretention() preserve it (re-sort after downsample, bisect-based truncation for retention).
2. Alert state transitions follow the state machine. OK → PENDING → ALERTING is the only firing path. PENDING resets to OK (not RESOLVED) if the condition clears before duration_seconds elapses. Only ALERTING → RESOLVED produces a RESOLVED alert.
3. Tag matching is a subset check. matchingkeys requires all tagsfilter key-value pairs to be present in the series tags, but extra tags in the series are fine. This means tagsfilter={"host": "web-01"} matches series with tags {"host": "web-01", "region": "us-east"}.
4. checkcondition uses the average of all points in the lookback window for threshold comparisons (gt/lt/etc.), not the latest point. The window is max(duration_seconds, 60).
5. Downsampling is lossy. It replaces raw points with bucket averages — min/max/percentile accuracy is lost for downsampled data.
Essentially none — this is interview-demonstration code. No validation on ingestion (negative timestamps, NaN values), no bounds checking on aggregation types, no protection against tag cardinality explosion. aggregate silently falls back to avg for unrecognized aggregation names. check_condition returns (False, 0) for empty data or unrecognized conditions, silently suppressing evaluation.
metrics-monitoring-and-alerting/test_metrics.py — How the alert state machine transitions are exercised, and edge cases around downsampling and retentionmetrics-monitoring-and-alerting/metrics.py:evaluate_alerts — The alert state machine is the most complex logic; trace the PENDING→ALERTING transition timing and the RESOLVED→re-fire pathpull-vs-push-alerting — This system uses poll-based evaluation (caller invokes evaluate_alerts); compare with push-based evaluation triggered on ingest, and the tradeoffs for latency vs. computational costtime-series-storage-tradeoffs — The sorted-list model has O(n) insertion; explore LSM-tree or columnar approaches (like Gorilla compression) used by real TSDB systemsmetrics-monitoring-and-alerting/plan.md — The design rationale and requirements that shaped this implementationmetrics-series-sorted-invariant — Every series in data is maintained in sorted timestamp order; ingest, downsample, and applyretention all preserve this invariantmetrics-alert-state-machine-four-states — Alert evaluation follows a four-state machine (OK → PENDING → ALERTING → RESOLVED) where PENDING requires duration_seconds to elapse before transitioning to ALERTINGmetrics-tag-matching-is-subset — matchingkeys performs subset matching on tags: a query with tags_filter={"a": 1} matches any series whose tags include a=1, regardless of other tags presentmetrics-condition-uses-window-average — Threshold alert conditions (gt, lt, gte, lte) compare against the average of all points in the lookback window, not the most recent valuemetrics-downsample-two-tier — Downsampling uses two age-based tiers: data 1–7 days old is compacted to 5-minute buckets, data older than 7 days is compacted to 1-hour buckets, both using averaging which loses min/max/percentile fidelity