Caching in System Design: Strategies, Eviction, Invalidation, and Stampede Prevention

Why Put a Cache Between the Application and the Database?

Every request that reaches the database pays the cost of retrieving data from the source. A database read might take 1–10 ms, while a cache hit in Redis costs approximately 0.1 ms—an order-of-magnitude difference before you account for database contention or network distance. Repeating those reads also consumes database connections and capacity that other requests need. Azure’s caching guidance describes the same scaling effect: serving repeated reads from a cache reduces latency and contention in the original data store.

The usual placement is straightforward: client → application server → cache → database. The application asks the cache for the requested object. If the object is present, the request is a cache hit and the application can return the cached copy. If it is absent, the request is a cache miss and the application must fetch the object from the database before it can respond and make that result available to later readers. Caching stores that copy closer to the reader so repeated fetches are faster and cheaper.

This works because access is not evenly distributed over time. Temporal locality means that after a user profile is read once, it is likely to be read again in the next few seconds. The same pattern makes a product page an obvious candidate: its description may change only once a week, yet customers may read it thousands of times per minute. Sending every one of those reads to the database is wasteful when most callers need the same value.

Caching’s job is to serve repeated reads from a nearby copy while accepting—and deliberately controlling—the correctness risk that the copy can become stale.

Cache-Aside: The Application Owns the Read Path

Cache-aside makes the application responsible for the read path. The application checks Redis before it queries Postgres; Redis does not know how to query the database or how to transform a database row. A present, unexpired value is a cache hit and can be returned immediately. A missing or expired value is a cache miss that the application must resolve.

The read sequence

For a user-profile service, use user:{id} as the cache key and store the profile as JSON with a TTL of 5 minutes. On a miss, the service reads the profile from Postgres, writes that result into Redis, and returns the same result to the caller. The cache therefore fills lazily: a cold service starts empty, and only profiles that are actually requested enter the cache. You do not need to pre-load every user profile.

function get_user_profile(id):
    key = "user:" + id

    cached_profile = check the cache for key
    if cached_profile exists:
        return the cached profile

    profile = query Postgres for the profile identified by id
    if the database read fails:
        return the service's database-error result

    write the serialized profile to Redis with a 5-minute TTL
    if the cache write fails:
        return the database result

    return the profile

The ordering matters. The service does not return until it has either obtained a value from Redis or obtained one from Postgres. If Postgres fails, there is no authoritative result to cache, so the request follows the service's database-error path and the cache remains unchanged. If Postgres succeeds but the Redis write fails, the database result is still valid for this request; return it and allow a later request to try the cache again. A cache write is an optimization, not a prerequisite for accepting a successful database read.

An eviction or TTL expiration is operationally the same as a miss at this read boundary: the next request pays for the database query and then attempts to repopulate Redis. That makes cache-aside graceful during cold start, but not free. The first request after a miss or eviction has database latency, and a traffic spike can cause many requests to discover the same missing key at once. Without coordination, each request can issue its own SELECT and cache update, creating the conditions for a cache stampede.

Worked Example: Caching a Frequently Read Product Page

Consider an e-commerce product page whose description changes once a week but is read thousands of times per minute. The description is a strong cache candidate: the source changes infrequently, while many requests ask for the same value. Without caching, every page view repeats the database read. With caching, the service retrieves the description once, stores a copy in Redis, and reuses that copy for subsequent reads. This is the core payoff of caching for repeatedly read data: fewer database operations and a faster read path.

Make the service-time difference explicit

Take a one-minute slice containing 1,000 product-page reads. If every read goes to the database, the aggregate service time is 1,000 reads × 1 ms, or 1,000 ms, at the low end of the database-read range. At 10 ms per read, it is 1,000 × 10 ms, or 10,000 ms. If those reads are cache hits, Redis takes approximately 0.1 ms per hit: 1,000 × 0.1 ms = 100 ms.

1000 * 1      # 1000 ms
1000 * 10     # 10000 ms
1000 * 0.1    # 100 ms

These are aggregate serialized service-time figures, not elapsed wall-clock time: requests can overlap across workers. The comparison still matters because sending all 1,000 reads to the database consumes 1,000–10,000 ms of database service time, whereas serving the same reads from Redis consumes approximately 100 ms of cache service time. The database therefore handles far fewer repeated reads, leaving its capacity for work that cannot be served from the cache. Azure’s caching guidance describes this pattern as most effective when data is read frequently, changes relatively infrequently, and the original store is slower than the cache; this product description has all three properties.

The arithmetic does not make the cached value authoritative. Redis holds a copy of the description captured at some point in the past. Because the product changes weekly, the design can tolerate some staleness only if the product requirement permits it; otherwise, a description update must trigger an explicit refresh or invalidation. The important design decision is not merely whether Redis is faster. It is how long readers may safely receive the previous copy before the service must obtain the changed description.

Choosing Cache-Aside, Write-Through, or Write-Behind

The choice is about which operation you make eager and which cost you accept. Cache-aside is lazy on reads: the application checks the cache, and on a miss it reads the database, populates the cache, and returns the result. Write-through is eager on writes: each write updates the cache and backing store synchronously before the response is sent. Write-behind is also eager at the cache, but defers the database update: it acknowledges after the cache write and flushes the change asynchronously.

The comparison people most often get wrong is treating write-through as a universal consistency fix. It is tempting because the write path waits for both destinations, so the cache and database do not drift simply because the application forgot to update one of them. But a concurrent read can still observe the old value momentarily during a write, and a failure between the two updates still needs a recovery policy. The decision flips on the value of durability and freshness versus write volume: use write-through when read-heavy traffic benefits from an immediately updated cache and the synchronous write cost is acceptable; avoid it or choose another path when updates are frequent and every write paying both costs is painful.

Eviction Is About Space; Invalidation Is About Correctness

Eviction and invalidation remove entries for different reasons. Cache eviction is space management: the cache has a fixed memory budget, and when it is full, a policy decides which entry to drop so another can fit. Cache invalidation is correctness: when the backing record changes, you decide that the cached copy is no longer valid and must be refreshed or evicted. Confusing the two makes incidents difficult to diagnose: memory pressure can remove a perfectly current value, while a cache with free memory can continue serving a value that the database has already changed.

Choose eviction from the access pattern

When the cache reaches its limit, LRU (Least Recently Used) evicts the entry that has not been touched for the longest time. It is a good default for many workloads because recent access is often a useful signal that an entry will be needed again. LFU (Least Frequently Used) evicts the entry accessed the fewest times, preserving items with durable popularity. That makes LFU a better fit when popularity is highly skewed—for example, viral content receives repeated requests while stale content is rarely touched. Redis documents approximate forms of these policies rather than requiring exact tracking; its LFU uses a probabilistic counter, while its LRU is approximate. Memcached uses a slab-based LRU.

A sequential scan exposes LRU’s blind spot: the scan touches each item once, making the scan look recent even if those entries will not be reused. As the scan fills the cache, it can push out genuinely hot data. LFU can resist that pattern when the hot entries have accumulated higher access counts, although historical popularity can be less useful when demand changes rapidly. In practice, combining a TTL with LRU gives each entry a freshness boundary while still providing a space policy when the memory budget is reached.

Choose invalidation from the data’s staleness tolerance

A TTL is the simplest invalidation strategy: attach a fixed lifetime to the cached copy, and remove it when that lifetime expires regardless of how often the entry was accessed. The trade-off is explicit: with a TTL of N seconds, users may see a value that is stale for up to N seconds, while the cache avoids needing to coordinate every database update. This works well with cache-aside, where the next miss fetches the current value and repopulates the cache.

When that bound is too wide, invalidate from the write path. After a database update, an event can tell the cache to refresh or evict the affected key. This is more precise than waiting for a TTL, but it creates a coordination problem: the database update and cache update are now two operations that must remain aligned. A delayed or failed cache update can still leave a stale copy, so event-driven invalidation exchanges TTL’s predictable staleness for more implementation complexity.

Make the choice per data type rather than applying one freshness rule everywhere. A follower count being two seconds stale may be acceptable; a bank balance being stale is not. Write-through also does not make every read continuously fresh: even when the cache and database are updated as part of the write, a concurrent read can momentarily observe the old value during that write. If the stale window is unacceptable, bypassing the cache or using a stronger coordination design may be preferable.

When Expiration Turns Into a Cache Stampede

A cache-aside design is not automatically safe under concurrency. The failure appears when a popular key expires while many callers are arriving: each request checks the cache, sees a miss, and independently starts the database read needed to regenerate the same value. The cache no longer absorbs traffic; it amplifies it. The operator sees a sudden rise in identical database queries, connection or query saturation, increased latency, and then errors as the database falls over.

Consider a news homepage whose top-10 article list is cached for 60 seconds. At second 60, the key expires. Thousands of users can miss together, and the database may see a sudden 10× spike from requests that all ask for the same top-10 list. A longer TTL changes when the failure happens, not the synchronized shape of the failure. The protection must spread or collapse regeneration work.

Three ways to prevent duplicate regeneration

Probabilistic early expiry starts refresh slightly before the TTL ends. During a refresh window, each request has a chance to become the refresher; the others continue using the current cached value. Randomizing the decision prevents all callers from choosing the same instant. For the news key, beginning the refresh window at second 58 instead of waiting for second 60 spreads regeneration before synchronized expiration. The cost is that some values are refreshed earlier than necessary and the cache may serve the old copy briefly while refresh runs.

A mutex on miss makes regeneration single-file for each key. The first caller that acquires the key's lock reads the database, repopulates the cache, and releases the lock. Concurrent callers wait; after the lock is released, they reread the cache and return the newly populated value instead of issuing duplicate database reads. The lock adds coordination and waiting latency, and a failed lock holder or failed database read needs an explicit error path.

function get(key):
  value = cache.get(key)
  if value exists:
    return value

  if try_lock(key):
    try:
      value = database.read(key)
      cache.put(key, value)
      return value
    except database_error:
      return regeneration_error
    finally:
      unlock(key)

  // Another caller owns the lock.
  wait for the lock to be released, then check the cache again
  value = cache.get(key)
  if value exists:
    return value
  return regeneration_error

The third option is background refresh. A job or request identifies keys that are about to expire and repopulates them before they become misses. Requests continue to use the current cached value while the refresh runs, so expiration does not create a regeneration cliff. This spends work on keys that may not be requested again, and it requires a scheduler or refresh worker plus a way to select the keys that need warming.

Cache invalidation earns its reputation as one of the two genuinely hard problems in computer science because correctness, timing, and coordination interact. A stampede is the timing-and-coordination version of the problem: the cached value becomes unavailable to many callers at once, and the backing store receives work that the cache was meant to prevent. Treat expiration as a concurrent event, not as a single request crossing a timer boundary.

Defending a Caching Design in a System Design Interview

A strong answer starts with the request path, not the product name: client → application → cache → database. The application checks the cache first. A hit returns the cached copy; a miss reaches the database, then the application or cache populates the cache before responding. State the workload before choosing a policy: is it read-heavy, write-heavy, popularity-skewed, or scan-heavy? That classification determines which trade-offs matter.

Make the trade-offs explicit

For lazy read population, choose cache-aside: the application loads only requested data. Choose write-through when the cache and database should be updated synchronously before acknowledging a write, and when write consistency matters more than write latency. Choose write-behind only when deferred persistence is acceptable and losing bounded, recent writes if the cache fails is an acceptable business outcome. It may fit a leaderboard; it is a poor choice for a financial transfer, where you should use write-through or skip the cache.

Then defend the memory policy against the access pattern. Prefer LFU over LRU when a small set of items is persistently and disproportionately popular. Be cautious with LRU under sequential scans: touching many items once can evict data that is otherwise hot. Ask how stale a value may be, and whether the system will use TTL—with an explicitly accepted staleness window—or event-driven invalidation when a database update must remove or refresh the copy promptly.

The operator’s question

The question that separates an operated design from a textbook answer is: what happens when a popular key expires under peak traffic? A good answer describes concurrent misses and names a guard: a mutex so one request regenerates the value, probabilistic early expiry so refresh begins before expiration, or background refresh that pre-warms expiring keys. It also asks what the database can tolerate while that refresh runs.