Designing a Pastebin: From Requirements to Bottlenecks

Requirements and scope

Start by fixing the product boundary. A paste is plain text, capped at 10 MB; binary and rich content remain open scope questions, and the MVP requires no authentication. Users can create, retrieve, and delete pastes through an explicit API Design: the client-server contract for operations, inputs, outputs, and failure cases. That contract must settle expiry and deletion behavior before you choose a database or storage service.

Functional requirements

The MVP accepts anonymous pastes, so there are no accounts or ownership records to authorize a delete. Instead, creation returns a deletion token; possession of that token grants deletion rights. Each paste receives a short identifier and an optional user-chosen TTL from 1 hour to forever. After the expiry time, the paste must not be served: distinguish an expired paste from an identifier that never existed, using the API's gone-versus-not-found contract. The system stores paste content in Object Storage, a flat key-value store for arbitrary blobs: the application writes content under a key and retrieves it by that key, without forcing the blob into a relational row.

The matrix makes the deliberate omissions visible. In particular, the MVP does not yet decide whether accounts, authenticated ownership, private pastes, a global deployment, or a precise cleanup schedule are required. Those are follow-up decisions, not assumptions hidden inside the design.

Non-functional requirements

Use a working scale of about 100 million stored pastes and a roughly 10:1 read-to-write ratio. Retrieval is the dominant path, with a read-latency target under 200 ms at p99. Make availability versus strong consistency an explicit design decision for ordinary reads; creation, deletion, and expiry semantics must still be explicit when metadata and content are handled by separate systems. The MVP keeps geography open rather than silently assuming either a single region or global replication.

This read-heavy shape makes Caching a first-class requirement, not a later optimization. Caching serves popular pastes from memory, absorbing hot-key traffic and protecting object storage and the metadata database. The cache must respect each paste's expiry rather than applying one lifetime to every entry. The next design step can therefore estimate the 10:1 traffic split, storage growth, identifier space, and the cache footprint before selecting capacity.

Capacity estimation

Traffic and storage

Start with the working assumptions: 10,000,000 new pastes per day, a 10:1 read-to-write ratio, and an average paste size of 10 KB. Convert the daily write volume to seconds: 10,000,000 pastes / 86,400 seconds ≈ 115 writes/second. Reads are ten times the writes: 115 × 10 ≈ 1,150 reads/second. The system is therefore read-heavy: reads outnumber writes by 10:1, so the read path—not paste creation—is the first path to size for.

For storage, multiply the daily paste count by the average content size: 10,000,000 × 10 KB = 100,000,000 KB = 100 GB/day. Across a year, 100 GB/day × 365 days ≈ 35 TB/year at planning precision. That is roughly 365 times the daily growth accumulated over a year, and it rules out treating paste bodies as ordinary relational metadata by default: the database would be carrying tens of terabytes of blob content alongside the small fields needed to find and expire each paste. Put the content in object storage and keep the metadata database focused on identifiers, timestamps, expiry, and storage keys.

ID space and collision handling

A seven-character identifier using a 62-character alphabet has 62^7 ≈ 3.5 trillion possible IDs. Against 1 billion existing pastes, the occupied fraction is 1,000,000,000 / 3,500,000,000,000 ≈ 0.00029, or about 0.03%—roughly one occupied ID in 3,500. A randomly generated candidate must still be checked for uniqueness before it is committed; on collision, generate another candidate and retry. The large space keeps the expected retry rate low without making uniqueness an assumption.

Size the cache for the hot slice

If the top 20% of pastes serve 80% of reads, target that hot slice rather than caching every paste. At 1,150 reads/second, the hot set accounts for 1,150 × 0.80 = 920 reads/second; the remaining pastes account for 1,150 × 0.20 = 230 reads/second. Caching the hot slice can therefore remove about 920 of 1,150 reads/second, leaving 230 backend reads/second—a 5× reduction in storage-facing read traffic. This is the capacity decision: provision the write path for about 115 writes/second, but protect object storage and metadata lookups from the much larger, concentrated read load.

API contract

API design is the contract between the client and server: it fixes the request shapes, response meanings, and failure behavior before you choose databases, caches, or object storage. For this MVP, keep the surface small: create a paste, retrieve it by short ID, and delete it with the secret issued during creation.

The contract below makes the creation inputs explicit. content is required; ttl and custom_alias are optional. A successful creation returns both the public short ID and a delete_token, because anonymous clients have no account identity to prove ownership later.

POST /pastes
Content-Type: application/json

{"content":"...","ttl":"...","custom_alias":"..."}

HTTP/1.1 201 Created
Content-Type: application/json

{"id":"...","delete_token":"..."}

HTTP/1.1 400 Bad Request

HTTP/1.1 <documented collision status>

GET /pastes/{id}
Accept: text/plain

HTTP/1.1 200 OK
Content-Type: text/plain

...

HTTP/1.1 404 Not Found

HTTP/1.1 410 Gone

DELETE /pastes/{id}
Content-Type: application/json

{"delete_token":"..."}

HTTP/1.1 <documented successful-deletion status>

Use POST for creation because the server allocates the paste ID and creates a new resource. A malformed body—such as missing content or an invalid TTL—gets 400 Bad Request. A syntactically valid request whose requested alias is already occupied receives the documented collision response. The creation response uses 201 Created, while a successful deletion has no response body and uses a status code selected and documented for this API.

Retrieval and expiry

GET /pastes/{id} returns the raw text with Content-Type: text/plain. Return 404 Not Found when the ID has never existed. Return 410 Gone when the paste existed but has expired or was deleted. That distinction gives clients a meaningful lifecycle signal and prevents an expired paste from looking like a spelling mistake.

Expiry is enforced at request time, not only by background cleanup. Once expires_at has passed, the server must return 410 and must not fetch, cache, or otherwise repopulate the paste. Deletion follows the same ownership rule: the caller submits the creation-time delete_token; the service verifies it before removing the paste. The token is the anonymous equivalent of an ownership credential, so it must be treated as a secret rather than included in the public ID.

Metadata and content model

Keep metadata and content on separate paths

Use a lean metadata record for everything you need to find, authorize, expire, or clean up. A Postgres implementation can use id as the primary key, retain the object-store reference in s3_key, and record created_at, nullable expires_at, and size_bytes. custom_alias is optional but must be unique when present. The row deliberately has no TEXT or BYTEA content column.

CREATE TABLE pastes (
    id          text PRIMARY KEY,
    custom_alias text,
    s3_key      text NOT NULL,
    created_at  TIMESTAMP NOT NULL,
    expires_at  TIMESTAMP NULL,
    size_bytes  bigint NOT NULL,
    UNIQUE (custom_alias)
);

DELETE FROM pastes
WHERE expires_at IS NOT NULL AND expires_at < NOW();

The lookup path is a point query by id; that primary key makes retrieval predictable without scanning paste content. Cleanup has a different access pattern: it finds rows where expires_at is non-null and earlier than the current time. Give that column an index or an equivalent expiry-oriented access path, then let a worker process expired rows in batches. A uniqueness constraint on custom_alias also turns alias collision detection into a database-enforced write check rather than a race-prone application convention.

The paste body belongs in S3-style object storage. Treat it as a flat key-to-value namespace: write the arbitrary blob under s3_key, then retrieve that value by key. The database can search lifecycle metadata, but it cannot search inside the opaque content without a separate indexing pipeline. This split lets the metadata store stay small and keeps content scaling independent from metadata queries.

Keys, partitioning, and query boundaries

Use a non-sequential seven-character base-62 id, or an equivalent collision-checked identifier, rather than an auto-increment value. Seven base-62 characters provide approximately 3.5 trillion possible keys, while avoiding a URL that reveals creation order and reducing the need for a single sequential allocator. If you choose random generation, enforce uniqueness and retry on collision; the uniqueness constraint makes the final check authoritative.

For Postgres, keep pastes keyed by id and use the expiry access path for cleanup. If the metadata volume later requires horizontal partitioning, hash-partition by id: it keeps point lookups distributed, but makes a global “all rows expiring before this time” query cross partitions. That trade-off is acceptable when cleanup workers process expiry ranges or maintain per-partition work queues. Do not pretend partitioning makes blob search possible—the content remains addressable only through its object key.

Components and request paths

Write path

The client sends a create request to the API servers. The API layer validates the paste, chooses or accepts its ID, and coordinates the two durable writes; it does not make Redis part of the creation path because Redis is a read cache, not the storage of record. First, the API server writes the content to object storage under an s3_key. It then writes the metadata row to the database, including id, s3_key, and expires_at. Only after both writes succeed does it return the short ID and deletion token.

That ordering gives reads a discoverable metadata record only after the content exists. If the object-store write succeeds but the database write fails, the API must not report a successful creation; it retries the metadata write or leaves the object for reconciliation. If the object-store write fails, it likewise does not create a usable metadata row. The cleanup worker handles remaining orphaned objects rather than making the client wait for a distributed transaction.

Read path

For GET /pastes/{id}, the API server checks Redis first. A hit returns the cached content only when its expiry has not passed. A miss goes to the metadata database to distinguish an unknown ID from an expired paste and to obtain the object-store key. An unknown ID returns 404; a known paste whose expires_at has passed returns 410. For an active paste, the API fetches the content from object storage, returns it, and populates Redis with a TTL clamped to the time remaining until expiry.

This places Redis directly in front of object-store reads, so repeated requests for a popular paste do not repeatedly consume object-store or metadata-database capacity. Redis is a disposable read cache rather than the source of record; define an explicit fallback policy for cache failure. If the metadata database is unavailable on a cache miss, the API cannot safely determine whether the ID is missing or expired; if object storage is unavailable, it cannot return an uncached body. Both cases should fail explicitly rather than manufacture a response. A cached, unexpired hit can still be served without contacting those dependencies.

Object storage and cleanup semantics

The storage boundary is a correctness and operations decision, not just a capacity decision. Keep paste content in object storage and keep the metadata database responsible for identity and lifecycle state. Object storage addresses arbitrary blobs through a flat namespace, so the database needs only short metadata such as id, s3_key, created_at, expires_at, and size_bytes. That lets content capacity and metadata capacity scale independently; it also leaves an attachment point for a CDN when popular, immutable objects need to be served closer to users.

Putting a 5 MB paste in a Postgres BYTEA or TEXT column couples every content read and write to the relational storage path. Large values displace useful metadata and indexes from the buffer pool, increase the data that replicas must copy, and enlarge point-in-time restore work. The problem is not that a relational database cannot store a blob; it is that millions of blobs make the database carry content, replication, backup, and metadata workloads together. Object storage is designed for unstructured objects and scale-out placement. Its durability comes from replication or erasure coding across failure domains, while its API-oriented access model and CDN integration trade some database-style queryability for cheaper, independently scalable blob storage.

Expiry therefore uses an ordered two-phase cleanup. First, a worker changes the metadata row to an expired state, or records that expires_at has passed. Only after that durable database transition does it request deletion of the corresponding S3 object. A GET for an ID that never had metadata returns 404; a GET whose metadata proves expiry returns 410 Gone. The distinction gives clients and operators different information without requiring the object to disappear at the exact expiry instant.

If the database is marked expired but S3 deletion fails or is delayed, subsequent reads still return 410, and a later cleanup attempt can reclaim the object. If S3 deletion succeeds before the database transition finishes—because a worker raced the normal ordering—the metadata row may still describe an active paste whose content is missing. The GET path must not treat that missing object as a new paste: it should fail closed, alert or enqueue reconciliation, and preserve the ID's existing metadata until the lifecycle worker resolves the inconsistency. Make object-deletion retries safe and keep the identifier reserved until metadata and cleanup reconciliation are complete.

This design accepts eventual physical cleanup in exchange for a simple visibility guarantee: once the database says expired, no read path serves the content. A sweeper can find expired metadata, enqueue purge work, and retry failures independently of user traffic. The same metadata key should identify the S3 object throughout its lifetime; reusing an ID while an old object or delayed cleanup still exists would turn an asynchronous failure into possible cross-paste content exposure.

Expiry-aware caching for hot pastes

A paste retrieval path should be cache-aside: the application decides when to read from Redis, when to consult metadata, and when to fetch the blob from object storage. That fits a read-heavy service because cold content is loaded only when requested, while popular content stays close to the API servers. It also avoids making every paste creation pay the cost of populating a cache entry that may never be read.

Write-through is not the natural fit here. It sends each write to both the cache and the database simultaneously, which keeps cached values fresher but adds write latency and creates a failure case when one write succeeds and the other fails. Paste creation should establish the durable metadata and content first; retrieval can populate the cache on demand.

GET /pastes/{id}

value = L1[id]
if value exists and value.expires_at > now:
    return value.content

value = GET short_key
if value exists and value.expires_at > now:
    return value.content

metadata = fetch metadata for id
if metadata does not exist:
    return 404
if metadata.expires_at is not null and metadata.expires_at <= now:
    return 410

if another request is loading id:
    wait for that request
    retry

content = fetch content from S3
if metadata.expires_at is null:
    cache_ttl = configured cache policy
else:
    cache_ttl = min(metadata.expires_at - now, max_cache_ttl)
store content in Redis with cache_ttl
return content

The important ordering is the expiry check. A hit is usable only while its stored expiry is later than the current time; otherwise the request continues through the durable path. On a miss, metadata distinguishes a never-existing paste from an expired one: return 404 for the former and 410 for the latter. For a live paste, clamp the cache lifetime to the smaller of the paste's remaining lifetime and max_cache_ttl. A paste that expires at time T must never remain readable from either cache after T.

Layering hot-key protection

A viral paste can overload each API server even when Redis is healthy: every request still crosses the network to the shared cache. An optional in-process L1 cache absorbs repeated requests locally, with Redis acting as L2 for misses. L1 reduces shared-cache traffic, but each API server now has its own copy. Keep its expiry no later than the paste expiry, and invalidate or bypass it when deletion or another content change makes the cached value unsafe. The tradeoff is lower latency and less Redis traffic in exchange for more copies to expire or invalidate.

Consider a paste that goes viral five minutes before its one-hour expiry. If you insert it with a fixed one-hour cache TTL, requests can continue receiving the cached content after the paste should produce 410 Gone. With an expiry-aware TTL, the entry receives only the remaining five minutes. At the expiry boundary, both L1 and Redis reject the value, and the metadata check returns the expired result instead of reviving the paste.

Collapsing cold misses

Expiry-aware TTLs prevent stale hits, but they can make many requests miss together when a hot entry expires. If every request independently fetches the same object from S3, the cache has merely moved the bottleneck to the origin. Use request coalescing, also called a single-flight pattern: the first request becomes the loader for that key, while concurrent requests wait and retry the cache after the loader stores the result. This bounds one burst to one origin fetch per key rather than one fetch per request. Probabilistic early expiry is another option: selected requests refresh before the deadline, spreading replenishment instead of allowing a synchronized expiry.

Anonymous deletion and short-ID safety

Anonymous pastes still need an owner-equivalent capability. At creation time, the service issues a deletion token and returns it alongside the short ID. The client must present that token to DELETE /pastes/{id}; the public ID alone is only a locator, not proof of ownership. Anyone who knows a paste URL can retrieve it in the MVP, but knowing that URL must not grant deletion rights.

Make deletion an explicit authorization check

The delete handler should resolve the public ID, validate the supplied deletion token against the token associated with that paste, and reject the request when they do not match. A successful deletion should make subsequent retrieval behave as deleted rather than silently recreating or replacing the paste. Keep the metadata uniqueness rule in force after deletion: if cleanup is asynchronous, a new paste must not reuse an identifier while an old record can still win a concurrent write.

Use a seven-character base-62 namespace for generated IDs. It contains approximately 3.5 trillion possible values; against the roughly 100 million stored pastes in this design, that is about 35,000 possible IDs per stored paste. The namespace is therefore ample, but randomness is not a uniqueness guarantee. Before publishing an ID, the service must make the identifier claim through the metadata store's uniqueness constraint. If the claim loses a race, generate another ID and retry. For a caller-supplied alias, do not retry with a different alias: return a collision response and leave the existing paste untouched.

This check must be atomic. Two create requests can both observe an alias as available if they perform a separate read followed by a write. Availability is only advisory; the uniqueness decision at insertion is authoritative. The winner persists the paste and receives its deletion token. The loser receives a collision response and must not receive a token for, or overwrite, the winner's paste.

Bottlenecks and scaling order

The first pressure point is the read path. At the estimated workload, 10M new pastes per day is about 115 writes per second, while a 10:1 read-to-write ratio produces about 1,150 reads per second—10 times as much traffic. Scale API servers horizontally first when request latency or API errors rise; this is the least disruptive fix, but it adds instances to operate and makes cache coordination more complex.

After API capacity, investigate concentration rather than average QPS. A viral paste can make one key much hotter than the aggregate suggests. Keep Redis as the L2 cache, add a small in-process L1 cache on each API server, and coalesce concurrent cold misses so one request fetches the paste from the origin while the others wait for that result. The cost is more memory, invalidation complexity, and coordination between requests.

The storage boundary is also a scaling decision. The estimated 100 GB of new content per day compounds to roughly 35 TB in a year—about 365 days of accumulation—so keep paste content in object storage and keep only short metadata in the relational database. This avoids making replication, backups, point-in-time restores, and metadata queries carry the blob volume. If cache misses or traffic spikes still drive object-store request and transfer pressure, extend the read path with a CDN; the tradeoff is additional transfer cost and another layer whose contents must be invalidated.

Later bottlenecks and deployment choices

When expiry checks and cleanup scans begin falling behind, index or partition expiry data and increase scheduler and cleanup-job capacity. Mark metadata expired, enqueue object deletion, and reconcile failures; the database and object store do not provide one atomic transaction, so the design can temporarily leave an orphaned object or metadata pointing at missing content. That failure is contained by retries and reconciliation, not eliminated.

Single-region deployment keeps consistency and operations simpler; global deployment improves geographic availability and latency but adds replication and consistency decisions. Choose deliberately whether an expired or deleted paste must disappear everywhere immediately, or whether higher availability permits a bounded convergence window. Each scaling step should follow an observed symptom: API saturation, a Redis hot key, origin request or egress growth, expiry-scan lag, then deletion backlog.