File: s3-object-storage/s3objectstorage.py

Date: 2026-06-05

Time: 14:01

s3-object-storage/s3objectstorage.py

Purpose

This file is a self-contained, in-memory simulation of Amazon S3's core API surface. It exists as a system design interview reference implementation — not a production storage engine, but a demonstration that the candidate understands S3's data model, versioning semantics, multipart upload protocol, presigned URL access control, and bucket policy evaluation. It owns the entire storage plane: bucket lifecycle, object CRUD, listing with prefix/delimiter pagination, copy, multipart upload assembly, presigned token generation/validation, storage class transitions, and policy checks.

Key Components

Data Classes

ObjectVersion — The atomic unit of storage. Every write produces one of these, whether the bucket is versioned or not. Notable fields:

Bucket — Container with four parallel dictionaries:

ObjectStorage Class

The public API class. Two class-level constants:

Bucket operations: createbucket, deletebucket, list_buckets

Object CRUD: putobject, getobject, deleteobject, headobject, copy_object

Listing: listobjects (prefix/delimiter/pagination), listobject_versions

Multipart: initiatemultipart → uploadpart (×N) → completemultipart | abortmultipart

Access control: generatepresignedurl, accesspresigned, setbucketpolicy, checkbucket_policy

Lifecycle: transitionstorageclass

Patterns

Version chain as append-only list. Each key maps to a list[ObjectVersion]. The latest version is always versions[-1]. In versioned buckets, putobject appends; in unversioned buckets, it replaces the entire list with a single-element list. This is the central design decision — it makes latest_version trivial (check the tail) and version listing a flat iteration.

Delete markers as sentinel versions. Deleting a versioned object doesn't remove data — it appends a zero-byte ObjectVersion with isdeletemarker=True. latestversion returns None when the tail is a delete marker, making the object appear gone to callers while preserving history. This mirrors real S3 semantics exactly.

Token-based presigned URLs. Rather than generating actual URLs, the implementation uses HMAC-SHA256 tokens stored in presigned dict. generatepresignedurl creates the token; accesspresigned validates it and delegates to get_object. This is a clean abstraction — the signing scheme is realistic while avoiding HTTP concerns.

Flat policy evaluation with short-circuit. checkbucketpolicy iterates policies in insertion order, returning on the first match. DENY wins over ALLOW when it matches first. Default is allow-all (no policies = open bucket, unmatched principal/action = allow).

Dependencies

Imports — all stdlib:

Imported by: tests3object_storage.py — the test suite is the only consumer.

Flow

Write path

putobject → validate bucket exists → coerce str to bytes → compute MD5 ETag → generate version ID (UUID if versioned, else None) → create ObjectVersion → append to version list (versioned) or replace list (unversioned) → return {versionid, etag, size}.

Read path

getobject → find bucket → if versionid specified, linear scan the version list; otherwise call latestversion which checks versions[-1] and rejects delete markers → return dict via versiontodict(includedata=True).

Delete path (versioned)

deleteobject → append a delete marker ObjectVersion to the version list → return {deleted, deletemarker, versionid}. The data is still there; only latest_version changes behavior.

Multipart upload

initiatemultipart → allocate uploadid, create empty parts dict → caller uploads parts in any order via uploadpart → completemultipart pops the parts dict, sorts by part number, concatenates bytes, delegates to put_object for storage. Abort cleans up both dicts.

Listing with prefix/delimiter

listobjects collects all keys matching the prefix, sorts them, then for each key: if a delimiter is present and appears in the key's suffix after the prefix, the key is folded into a common prefix (simulating S3's "virtual directory" behavior) and skipped. Otherwise it's added to the results list up to maxkeys. Pagination uses the last returned key as continuation_token.

Invariants

1. Bucket names must match ^[a-z0-9][a-z0-9-]{1,61}[a-z0-9]$ — enforced at create_bucket.

2. Object keys must be ≤1024 characters — enforced at put_object.

3. Bucket deletion requires the bucket to be logically empty: no live objects, and in versioned buckets, no non-marker versions at all.

4. Version lists are never empty while the key exists in bucket.objects. If a non-versioned delete removes the only version, the key is deleted from the dict entirely.

5. Multipart uploads must have at least one part — complete_multipart raises ValueError on empty parts.

6. Parts are assembled in sorted part-number order regardless of upload order.

7. Presigned tokens expire — accesspresigned checks expiresat against current time.

8. SECRET is class-level, not instance-level — all ObjectStorage instances in the same process share the same signing key, meaning presigned tokens from one instance are theoretically valid on another (though presigned dict is instance-scoped, so this doesn't actually work cross-instance).

Error Handling

The module uses a straightforward exception strategy with no custom exception types:

Topics to Explore

Beliefs