Date: 2026-06-05
Time: 13:45
design-youtube/design_youtube.pyThis file implements a simulation of YouTube's core backend subsystems as a system design interview exercise. It doesn't build a real video platform — it models the *architectural concepts* you'd discuss in an interview: video upload pipelines, adaptive streaming, approximate counting at scale, and multi-strategy recommendation. Each component is self-contained and testable in-process without any external infrastructure.
Video — The central entity. Tracks metadata (title, tags, uploader), lifecycle status via VideoStatus, engagement counters (views, likes, dislikes), and post-processing artifacts (manifest, thumbnails). Status transitions are UPLOADING → PROCESSING → READY | FAILED.
TranscodedVariant — Represents a single encoding of a video at a specific resolution/bitrate. Created during transcoding, consumed by StreamingManifest.
StreamingManifest — Models an HLS/DASH-style adaptive bitrate manifest. selectquality(bandwidthkbps) walks variants from highest to lowest bitrate and returns the best one the client can sustain. Returns None if no variant fits — the caller must handle that.
ProcessingDAG — A general-purpose DAG executor. Stages are registered with named dependencies, then executed in topological order (Kahn's algorithm). Two key behaviors:
ValueError.SKIPPED — failure cascades through the graph.VideoUploadPipeline — Wires up a specific 5-stage DAG for video processing:
validate → transcode ─┐
validate → thumbnail ──┤→ finalize
validate → metadata ───┘
transcode, thumbnail, and metadata are independent after validation — they'd run in parallel in a real system. The finalize stage fans in, assembling the manifest and thumbnails onto the Video object.
The pipeline handlers are plain functions (handlevalidate, handletranscode, etc.) that read/write a shared ctx dict. This is essentially a poor-man's blackboard pattern.
MorrisCounter — Implements Morris's approximate counting algorithm (Morris+ variant). Uses 32 independent counters, each incremented with probability 1/2^c. The estimate 2^c - 1 is averaged across all counters for better accuracy. This is relevant at YouTube scale where exact counting of billions of views is expensive.
HyperLogLogCounter — Estimates cardinality (distinct count) using HyperLogLog with configurable precision (default p=14, so m=16384 registers). Includes the small-range correction from the original Flajolet paper. Used here to approximate unique viewer counts.
ViewCounter — Facade that combines all three counting strategies per video:
VideoStore — In-memory CRUD store. search() does case-insensitive substring matching on title and tags, filtering to READY or UPLOADING status. upload() generates a UUID and returns the Video object in UPLOADING state — the caller is responsible for running the pipeline separately.
RecommendationEngine — Three strategies, combinable via weighted scoring:
1. Content-based (recommendbycontent): Jaccard similarity on tag sets. Simple and deterministic.
2. Popularity (recommend_popular): Sorts by exact view count.
3. Collaborative filtering (recommend_collaborative): Co-occurrence — "users who watched video X also watched Y." Counts how many co-watchers viewed each unseen video.
get_feed merges all three using reciprocal rank fusion: each strategy contributes weight * 1/(rank+1) to a video's score. Already-watched videos are excluded. Default weights favor collaborative (0.5) over popular (0.3) over content (0.2).
ProcessingDAG (generic executor) and VideoUploadPipeline (specific wiring) keeps the engine reusable.ctx dict rather than return values. This avoids coupling stages to each other's types but sacrifices type safety.ViewCounter layering exact + approximate is a common interview talking point.get_feed that blends them with configurable weights.failure_rate on ProcessingStage enables testing failure cascades without mocking — the pipeline can simulate real-world transcoding failures.Imports: All stdlib — hashlib (HLL hashing), math (HLL correction), random (Morris counters, fault injection), uuid (video IDs), collections.defaultdict, dataclasses, enum, typing.
Imported by: testdesignyoutube.py — the test suite exercises all components.
No external dependencies. No cross-module imports within the repo.
A typical lifecycle:
1. VideoStore.upload() creates a Video in UPLOADING status.
2. VideoUploadPipeline.process() sets status to PROCESSING, builds the DAG, executes it.
3. DAG runs: validate → (transcode | thumbnail | metadata) → finalize.
4. handle_finalize builds the StreamingManifest and sets status to READY.
5. Views are recorded via ViewCounter.record_view(), which updates exact counts, Morris counters, HLL, and watch percentages.
6. RecommendationEngine.get_feed() blends strategies to produce a ranked list for a user.
handlevalidate rejects formats not in VALIDFORMATS (mp4, avi, mkv, mov, webm) and durations exceeding 43200s (12 hours).detectcycle() enforces no cycles before execution. This is checked on every execute() call.SKIPPED. The pipeline sets VideoStatus.FAILED if finalize doesn't complete.record_view clamps to [0.0, 100.0].select_quality always returns the highest-bitrate variant that fits, or None. No randomness.ProcessingDAG.execute() — the bare except Exception swallows the error, marks the stage FAILED, and lets execution continue for independent branches. This means stage errors never propagate to the caller; you only see them in the returned status dict.handle_validate) raise ValueError, which the DAG catches.VideoStore.get() returns None for missing videos rather than raising.VideoStore.delete() returns a boolean success flag.StreamingManifest.select_quality() returns None when no variant fits the bandwidth — callers must check.design-youtube/testdesignyoutube.py — See how the pipeline, counters, and recommendation engine are exercised; reveals edge cases the implementation handlesdesign_youtube.py:ProcessingDAG.execute — Trace how failure cascades work through the DAG; consider what happens when thumbnail fails but transcode succeedsmorris-counter-accuracy — Compare Morris counter estimates against exact counts at various scales to understand the precision/memory tradeoffhll-small-range-correction — The HLL implementation includes the linear counting correction for small cardinalities; worth understanding when it activates and whydesignyoutube.py:RecommendationEngine.getfeed — Explore how reciprocal rank fusion blends strategies and how weight tuning affects feed compositiondag-failure-cascade — When a ProcessingDAG stage fails, all transitively dependent stages are marked SKIPPED; independent branches continue executingpipeline-status-contract — VideoUploadPipeline sets video status to FAILED if and only if the finalize stage does not reach COMPLETEDhll-default-precision — HyperLogLogCounter defaults to precision 14 (16384 registers), matching the standard HLL recommendation for ~1% error raterecommendation-excludes-watched — get_feed excludes videos the user has already watched from popular and content-based scores, but collaborative filtering excludes them in its own co-occurrence logicsearch-status-filter — VideoStore.search returns only videos with status READY or UPLOADING, excluding PROCESSING and FAILED videos