Design a URL Shortener: From Requirements to Bottlenecks

Requirements: Define the Contract Before Choosing Components

Start by fixing the contract, not by naming a database. The core product has two actions: create a mapping from a long URL to a short code, and resolve that code back to the destination. Before drawing the request path, ask whether users may choose custom aliases, set an expiration time, or delete links. Also decide whether shortening the same long URL twice must be idempotent—returning the existing code—or may create two independent codes. These choices change uniqueness, ownership, and cleanup requirements.

Keep product behavior separate from operational targets. For redirects, set an explicit latency and availability objective; a reasonable starting point is p99 under 50 milliseconds end to end and 99.99% availability. Analytics is a separate requirement: collect click events without making the redirect wait for an analytics write. If analytics may lag by seconds, an asynchronous pipeline is acceptable; if every click must be counted synchronously, the redirect path has a different failure and latency budget.

The scale assumption is the constraint behind every later component choice. Ask how many URLs are shortened per day and how often those codes are read. Use approximately 100 million shortens per day and a read/write ratio of at least 10:1 as the baseline. A concrete interview assumption is 100 million daily shortens and 1 billion daily redirects: that is roughly 1,200 sustained writes/s and 12,000 sustained reads/s; with a 2× spike factor, that is roughly 2,400 peak writes/s and 24,000 peak reads/s. The read-heavy shape makes redirect availability and latency more important than optimizing the creation path first.

Estimation: Turn Product Assumptions Into Capacity

Before choosing a database or cache, turn the product assumptions into rates. Assume 100 M shortens per day and 1 B redirects per day. Redirects therefore outnumber shortens by 10:1, so the redirect path—not the write path—sets the first throughput target.

Traffic

• Shortens: 100,000,000 URLs/day ÷ 86,400 seconds/day = 1,157 writes/s, which rounds to approximately 1,200 sustained writes/s. With the required 2× peak factor: 1,200 × 2 = 2,400 peak writes/s—twice the sustained rate.
• Redirects: 1,000,000,000 redirects/day ÷ 86,400 seconds/day = 11,574 reads/s, which rounds to approximately 12,000 sustained reads/s. At the same 2× peak factor: 12,000 × 2 = 24,000 peak reads/s—twice the sustained rate.
• The resulting peak mix is approximately 24,000 reads/s versus 2,400 writes/s, still a 10:1 read-to-write ratio. Your redirect service and its lookup tier must therefore absorb roughly ten times as many operations as the shortening path.

Storage

• Rows: 100,000,000 rows/day × 365 days/year × 5 years = 182,500,000,000 rows, or approximately 180 B rows over five years.
• Raw storage: 182,500,000,000 rows × 500 bytes/row = 91,250,000,000,000 bytes, or approximately 90 TB raw. The 500-byte estimate includes the seven-character short code, a long URL of up to 2 KB, timestamps, and a user ID; the row count is the larger multiplier by far.
• This is raw storage, before replication, indexes, and backups. The important comparison is scale: you are retaining roughly 180 B rows and approximately 90 TB, not merely serving 24,000 peak lookups per second. That storage footprint forces an explicit decision about whether a relational primary with read replicas remains appropriate or whether you need a distributed key-value or wide-column store.

Redirect bandwidth and cache

• Redirect bandwidth: 12,000 redirects/s × 200 bytes/response = 2,400,000 bytes/s, or approximately 2.4 MB/s outbound. Each request contributes only about 200 bytes, so the payload is small relative to the 12,000 requests/s it accompanies: this is primarily a latency and request-throughput problem, not a bandwidth problem.
• Hot cache: if the top 20% of URLs produce 80% of traffic, caching the assumed 20 M hot codes at 500 bytes/code requires 20,000,000 × 500 = 10,000,000,000 bytes, or approximately 10 GB. That assumed hot set is about 10 GB ÷ 90 TB = 1/9,000, roughly 0.01% of the five-year raw database footprint, while covering 80% of requests under the stated concentration assumption.

API Design: Keep the Surface Small and Make Redirect Semantics Explicit

Two endpoints, two jobs

Keep the public contract small: one operation creates a mapping, and one operation consumes it. POST /shorten is the right shape for creation because the service allocates the short code and persists a new relationship. The request accepts the required long_url plus optional ttl and alias; the response returns the allocated short_code. A custom alias still goes through the same creation path, so it must compete for the same short-code namespace and uniqueness rules.

POST /shorten
Content-Type: application/json

{
  "long_url": "<long URL>",
  "ttl": "<optional expiration time>",
  "alias": "<optional custom alias>"
}

Response
Content-Type: application/json

{
  "short_code": "<short code>"
}

GET /<short code>

HTTP/1.1 302 Found
Location: <long URL>

# Redirect choice: use 302 when analytics are needed; consider 301 when the destination is permanent and analytics are not needed.
# After returning the redirect, fire an async analytics event.

The redirect endpoint is a GET because the client supplies the short code and expects the service to resolve it, not create another resource. The response's Location header carries the destination. Use 302 Found when the service must observe redirects: the request reaches the redirect service, which can emit a click event asynchronously without making the user wait for analytics storage. That keeps click recording off the latency-critical response path.

This choice is a product decision, not a cosmetic status-code change. If a fraud-detection pipeline must observe every click, return 302 and publish the click event to an asynchronous downstream pipeline after issuing the redirect. A 301 may make repeat visits bypass the service entirely, so no downstream consumer can infer that every click was observed. If analytics are not required and the destination is genuinely permanent, 301 can reduce repeat traffic through the service, but it also gives up that control.

Data Model: Make the Short-Code Lookup the Fast Path

The mapping table should serve the redirect path first: given one short_code, return one destination without scanning unrelated mappings. Keep the record deliberately narrow: a roughly 7-character code, a long URL up to 2 KB, creation and expiry timestamps, and the creating user's ID. The short code is the identity of the mapping; the long URL is an alternate lookup only when the product promises idempotent shortening.

Define a mapping record keyed by a unique short_code, with the long URL, creation and expiry timestamps, and the creating user's ID. If idempotent shortening is required, enforce uniqueness for long_url in the chosen database; otherwise do not add that uniqueness constraint.

The unique short_code key makes the redirect lookup a single-column point read rather than a scan of unrelated mappings. A uniqueness constraint for long_url is optional from a product perspective: keep it when the same long URL must return the same code, and omit it when every shorten request may create a new mapping. Store creation and expiry timestamps for lifecycle checks, and store the creating user's ID when ownership is in scope.

Uniqueness and idempotent shortening

Generated codes and custom aliases must share the same uniqueness boundary. Store both in short_code; otherwise an alias can be accepted even though it collides with a generated code, and the redirect path has no unambiguous row to return. When idempotent shortening is required, look up the existing mapping by long URL and enforce uniqueness when creating the mapping. If two requests race, the uniqueness constraint—not an application-level check alone—must decide which mapping wins; the losing request retries its lookup and returns the committed mapping.

Store and partition choice

At the stated retention horizon, 100 million mappings per day over five years is about 180 billion rows; at approximately 500 bytes per row, that is about 90 TB of raw mapping data. Pair that footprint with roughly 12,000 redirect reads per second sustained and roughly 24,000 at the stated 2× peak. PostgreSQL with a unique indexed short_code and read replicas is a defensible starting point when its operational limits and replication lag are acceptable. A key-value or wide-column store becomes attractive when the point-read workload and dataset no longer fit the chosen relational topology; the numbers justify the comparison, not an automatic migration.

Do not shard merely because the table is large. Begin with one primary and replicas if that topology can absorb the write rate, index maintenance, storage, and recovery requirements. When one primary, its indexes, or replica capacity becomes the limiting resource, consider partitioning by short_code so a redirect point read can be routed by its lookup key. A query organized by created_at cannot be answered from one such partition and requires a fan-out across partitions. The choice therefore favors the dominant access pattern over convenient operational reports.

High-Level Design: Separate the Write Path From the Redirect Path

The design has two request paths with different bottlenecks. Shortening is a write: validate the request, allocate a code, and persist the mapping. Redirecting is a high-volume point read: resolve the code with the lowest possible latency. Put both behind a load balancer, but keep the application workers stateless so you can add or remove workers without moving user state.

On the shorten path, the client sends POST /shorten to the load balancer. A stateless shorten worker generates the code and writes the mapping to the database primary. After the primary confirms the write, the worker can pre-warm Redis with the new mapping; lazy population is also valid when the link is unlikely to be used immediately. The database primary owns writes so code uniqueness and expiry metadata have one authoritative write path.

On the redirect path, the client sends GET /{short_code} through the same load balancer to a stateless redirect worker. The worker checks Redis first. A cache hit returns the destination without touching the database. On a miss, it reads from a database replica, stores the result in Redis, and returns the redirect. Replicas keep the roughly 12,000 sustained reads per second away from the primary while the cache absorbs repeated requests; the estimated 90 TB raw database is also far larger than the roughly 10 GB hot working set, so caching avoids treating the whole corpus as a hot read workload.

The estimates determine how far you need to go. A read-heavy workload—about 1 billion redirects per day versus 100 million shortens, or roughly a 10:1 read-to-write ratio—justifies Redis and read replicas before database sharding. Start by measuring whether an indexed PostgreSQL primary with replicas meets the latency and throughput target. If the 90 TB, high-read design exceeds the storage, replication, or operational limits of that choice, evaluate a key-value or wide-column store and explain which limit it removes. The decision should follow the measured workload, not the database brand.

Short-Code Generation: Uniqueness Without a Global Write Hotspot

Short-code generation has two separate requirements: the code must be compact enough for a URL, and its uniqueness must survive concurrent writes from many application servers. Two design families are useful: derive the code from the long URL, or allocate a unique integer and encode it.

Hashing: compact, independent, but collision-prone

A hash-based design canonicalizes the long URL, computes an MD5 or SHA-256 digest, and encodes a truncated result with base62. With seven characters, the code space is 62^7, approximately 3.5 trillion combinations. That is a large space, but truncation means two different URLs can still produce the same seven-character code; a hash is not a uniqueness guarantee.

The write must therefore make the short code unique at the database boundary, not only in application memory. Insert the mapping under a unique constraint on short_code. If the insert reports a conflict, look up the existing mapping: return it when it belongs to the same long URL, preserving idempotent shorten behavior; otherwise derive a new candidate, such as by adding a salt and hashing again, then retry. A suffix-and-rehash strategy is another valid collision-resolution path, but every retry must still pass through the same uniqueness check.

HASH CREATION

create_short_url(long_url):
    candidate = base62(truncate(MD5(long_url)))[0:7]

    loop:
        result = INSERT mapping(long_url, short_code = candidate)
        if result succeeds:
            return candidate
        if result is a UNIQUE constraint violation:
            existing = find mapping by short_code = candidate
            if existing.long_url == long_url:
                return existing.short_code       // idempotent mapping
            candidate = base62(truncate(SHA-256(long_url + salt)))[0:7]
            retry

COUNTER-RANGE ALLOCATION

server:
    if local_range is exhausted:
        local_range = request the next range of 1 M IDs
    id = next unused ID in local_range
    local_range advances
    short_code = base62(id)
    persist mapping(short_code, long_url)

    if server crashes before consuming local_range:
        the unconsumed IDs become a lost-range gap
        request another range after restart

The important failure distinction is between a duplicate request and a true collision. The former should return the existing mapping instead of creating another code. The latter should never overwrite the existing owner of the code. This turns a probabilistic generator into a safe write path, at the cost of an extra lookup and retry only when the candidate is already occupied.

Counters: uniqueness by construction, coordination by design

A global atomic counter takes the opposite approach. A counter service assigns the next integer, and the application base62-encodes it. Redis's INCR is one example of the allocation operation. This is fast and collision-free, but every shorten request depends on the counter service: it is a write bottleneck and a single point of failure. Sequential IDs also expose creation order and make enumeration easier.

Range allocation removes that central operation from every request. A ticket service hands each application server a disjoint range, such as 1 M IDs. The server consumes IDs locally, base62-encodes each one, and requests another range only after exhausting its current range. If it crashes midway, the unused IDs in that range become a gap. That gap is acceptable: uniqueness matters, not contiguous numbering. Base62 makes the allocated value compact, but it does not hide its sequential nature.

This is the practical compromise for many designs: the allocator coordinates ranges rather than individual writes, while application servers generate codes locally without collisions. Custom aliases must enter the same uniqueness boundary. Store custom aliases under the same unique index or namespace as generated codes, and reject conflicts. Without that check, users can squat on codes or race a generated mapping for the same visible path.

Caching and Redirect Performance: Protect the Database on the Hot Path

Cache-aside on the redirect path

Treat the short-code mapping as the cache value: the code is the key, and the long URL plus its expiry and deletion state are the value. A redirect worker checks Redis first. On a hit, it returns the redirect without consulting the database. On a miss, it reads the mapping from the database, stores that mapping in Redis with an expiry aligned to the URL's own expiry, and returns the redirect. This keeps the database off the normal read path while preserving it as the source of truth.

Permanent URLs can use long cache lifetimes, but a permanent cache entry is safe only while the destination and deletion state cannot change. If either can change, active invalidation must remove or replace the cached mapping when the write occurs. Expiring URLs need a cache lifetime that does not outlive the mapping; otherwise Redis can redirect to a URL that the product considers expired. LRU eviction is a natural fit for the remaining capacity: codes that stop receiving traffic become cold and age out, preserving space for links that are being clicked.

Population and the viral-link failure mode

Lazy population is the simpler write policy: shortening writes the mapping to the database, and the first redirect fills Redis. Write-through or explicit pre-warming also writes the new mapping to Redis during shortening. That adds write-path coordination, but it removes the first-hit penalty for a link you already expect to be popular. For an ordinary link, lazy population avoids work that may never be needed; for a campaign link, pre-warming can be justified by the product context.

Consider a marketing link created 10 seconds before a television advertisement airs. The first viewers arrive together while the cache is cold. Without coordination, thousands of workers observe the same miss and issue the same database read. The database receives a burst for one value even though one read could have populated the entire hot entry. Protect the refill with a per-key mutex or semaphore: one worker owns the refill, while the others wait briefly or retry the cache. Probabilistic early expiry can spread refills before the entry expires, reducing the chance that many requests discover an empty key at once.

Cold start after a Redis restart

A Redis restart removes the working set, so the system must treat recovery as a controlled warm-up rather than allowing every miss through. Request coalescing still permits only one refill for a given short code. Replicas provide additional database capacity for fallback reads, and warm-up should prioritize known hot codes instead of replaying the entire keyspace. During recovery, the redirect service should keep serving cache hits, route misses through the refill guard, and apply its fallback behavior when the database or cache is unavailable rather than multiplying retries against the primary.

The checkpoint scenario makes the scale problem explicit: 50,000 redirects per second after a cache loss is roughly four times the 12,000-read-per-second sustained estimate. If all 50,000 requests independently miss, the database sees the burst; if requests for the same viral code coalesce, one read repopulates the entry and the remaining requests wait or retry. After the cache is hot, the redirect path returns to serving the repeated mapping from memory. The operational questions are whether the hit rate recovers, how long warm-up takes, and whether database fallback remains bounded during that interval.

Bottlenecks and Scaling: Identify the Next Limit Before You Add a New Tier

Scale the system by the constraint that is actually closest to failure, not by adding the most sophisticated storage tier. At the estimated rate, redirects are roughly 10× writes: about 12,000 reads/s versus 1,200 writes/s sustained, with peak planning roughly doubling both. Redis and read replicas therefore absorb redirect pressure; the single database primary is the first write-side constraint as shorten traffic rises.

A shorten service reaching 5,000 writes/s on one PostgreSQL primary is handling roughly four times the estimated sustained write rate. Before changing databases, reduce contention where the write path permits it: batch writes, scale the primary vertically, and let application servers pre-fetch ID ranges instead of requesting every identifier from one global counter. Range allocation costs some operational complexity and can strand unused IDs when a server crashes, but it removes per-request counter contention.

The decision grid orders the first response before the structural migration. The important escalation is from an indexed PostgreSQL primary with replicas, to partitioning or sharding when the footprint and write budget justify it, and only then toward a KV or wide-column store when the read-heavy mapping path keeps outgrowing the relational setup. The five-year estimate is approximately 90 TB: 180 billion rows at about 500 bytes each. That is not a reason to abandon PostgreSQL immediately, but it is a reason to measure partition size, replica lag, and cache effectiveness from the start.

A single global auto-increment counter becomes shard-hostile because every shard still depends on one allocation point. It also exposes creation order. If multiple database nodes are required, partitioning or sharding the mapping store removes the primary's storage and write ceiling, but adds routing, rebalancing, and failure-recovery work. Moving ID generation to a counter service removes identifier allocation from the database, but creates another highly available dependency; range allocation is the less centralised intermediate step.

The remaining limits are consistency and concentration rather than raw component count. Replica lag can make a newly created code invisible on a redirect miss, so route freshness-sensitive misses to the primary while keeping ordinary reads behind the cache. Partitioning by an access-aligned key or migrating incrementally to a KV or wide-column store reduces pressure, but each step increases operational complexity and migration risk. The design's unavoidable failure is a simultaneous cache loss during a viral event: without stampede protection and enough authoritative capacity, no database choice makes that burst free.