Latency, Throughput, and the Numbers That Shape System Design

“Fast” Describes Two Different Things

“Fast” is ambiguous in a design review. A request can finish quickly while the system handles only a small number of requests, or the system can complete many requests per second while each request waits in a queue. If you optimize only the first property, you may leave capacity unused; if you optimize only the second, you can make individual users wait longer.

Consider a single-threaded HTTP server handling 1,000 requests per second at 5 ms p99. Switching to 100 threads can raise throughput to 20,000 requests per second—20 times as much—but p99 can reach 50 ms, 10 times higher, because the threads contend for the same database connection pool. The system is better at absorbing work, but worse at completing one request promptly.

The distinction changes what you measure and what you optimize. A user-facing endpoint usually needs a bounded response time; a batch or ingestion pipeline may need sustained capacity. Treating a capacity problem as a latency problem leads you to shave time from individual operations without removing the shared bottleneck. Treating a latency problem as a throughput problem leads you to add concurrency and may worsen the queue that users are already waiting in.

Latency and throughput are therefore separate design dimensions that interact under load, not two names for the same notion of speed. The next mechanism gives each dimension its own measurement so you can budget response time without losing sight of system capacity.

Latency: One Operation, One Clock

Latency is the elapsed time for one operation, measured from the moment the request bytes leave the sender until the response is fully received. That clock includes every hop on the critical path: transmission, propagation, server-side queuing, computation, I/O, and the return path. A request is not complete when the server begins processing it; it is complete when the caller has the response.

Separate the clock into two useful categories. Propagation latency is the time required for signals to travel through physical distance. It is constrained by the speed of light, so moving a dependency closer is the architectural lever; tuning application code cannot remove that distance. Processing latency is everything your system adds while handling the request: waiting in a queue, running computation, reading storage, or serializing the response. A US-East to US-West round trip has approximately 60 ms of propagation latency imposed by physics alone. Every millisecond above that propagation floor comes from additional processing, queuing, transmission, or network-path latency.

In production, report latency as a percentile rather than only an average. p50 is the median request: half complete faster and half slower. p99 is the point below which 99% of requests complete, leaving 1 in 100 beyond it. p999 exposes an even deeper tail. The average can look healthy while a small group waits far longer, because fast requests pull the mean down. For an SLA, that tail is not an outlier you can ignore; it is the experience of a predictable fraction of users.

Use the split to diagnose the right class of problem. A higher propagation component points to distance or the number of network crossings. A higher processing component points to queueing, compute, I/O, or work added inside the request path. The same endpoint can therefore have a good median and a bad tail when contention occasionally makes processing wait, while a remote dependency can impose a floor that no amount of local optimization can cross.

Throughput: Sustained Capacity, Not a Burst Peak

Throughput is the number of operations a system completes per unit of time: requests per second, records per second, or bytes per second. Measure it under a sustained, steady load, not from the highest short burst the system can absorb. A burst peak tells you how much work fits temporarily in buffers; sustained throughput tells you how much work the system can keep completing without the queue growing.

Trace each request through the resources it consumes. CPU work, network transfer, and disk I/O may all contribute, but the slowest shared resource establishes the ceiling. If a disk serves 500 MB/s and each query scans 5 MB, the disk can support 100 QPS: 500 divided by 5. A faster CPU cannot raise that limit because every additional query still needs the same disk service.

The calculation is deliberately simple: name the resource's service rate, name the work required by one operation, and divide the first by the second.

The result is 100 QPS, not 500 QPS: each query consumes 5 MB of the disk's 500 MB/s budget. That ceiling applies before application overhead, contention, or other disk users consume capacity, so it is an upper bound rather than a performance promise.

Concurrency raises capacity—and queues

Adding workers can increase throughput when the original worker was idle or was the limiting resource. It can also increase per-request latency: more requests reach the same database connection pool or disk, compete for service, and wait in a longer queue. The system may complete more work per second while each individual request takes longer.

A concrete case makes the trade-off visible. A single-threaded HTTP server handles 1,000 requests/s at 5 ms p99. Moving to 100 threads raises throughput to 20,000 requests/s—a 20× increase—but p99 rises to 50 ms, 10× higher, because the threads contend for the same database connection pool. The extra threads improved concurrency, not the pool's underlying service capacity.

Jeff Dean’s Latency Numbers: A Ruler for Architecture

Jeff Dean’s latency numbers are a cost ladder, not a set of trivia questions. They give you an order-of-magnitude estimate before you have a benchmark: an L1 cache reference is about 1 ns, main memory about 100 ns, a random SSD read about 100 µs, a spinning-disk seek about 10 ms, and a cross-datacenter round trip about 150 ms. The values vary with hardware and workload; use them as a ruler for architecture, then replace them with measurements for your system.

The ratios are the useful part. Main memory is roughly 1,000× faster than a random SSD read, and an SSD read is roughly 100× faster than a spinning-disk seek. A same-datacenter round trip of about 500 µs is roughly 500,000 1-ns L1-cache references. That is why replacing one remote hop with a local lookup can matter more than shaving a few instructions from application code. The exact ratio depends on which cache level, storage operation, and network path you compare.

Use the ladder by translating each operation into the budget it consumes. Ten sequential SSD reads at roughly 100 µs each cost about 1 ms; ten disk seeks at roughly 10 ms each cost about 100 ms. Conversely, one cross-datacenter round trip can consume an entire tight request budget. If your SLA is 100 ms at p99, a required cross-datacenter call costing about 150 ms has already exceeded it before application logic, serialization, or data access runs. Put that data closer to the caller or serve it from a local cache instead of trying to optimize code after the remote call.

Worked Example: Budget a Feed Endpoint Before You Build It

Start with the SLA

Treat the 200 ms p99 SLA as a budget, not as a number you check after implementation. Every dependency on the request's critical path spends part of that budget. The feed-render endpoint needs three results: a user profile from another datacenter, ranked posts from a local SSD, and advertisements from a service in the same datacenter. Their stated costs are approximately 150 ms, 5 ms, and 2 ms.

If the endpoint calls them sequentially, the arithmetic is direct: 150 ms + 5 ms + 2 ms = 157 ms. That leaves 200 ms − 157 ms = 43 ms for request handling, serialization, and any other work. The design fits on paper, but the margin is small: the known dependency cost consumes 157 / 200 = 78.5%, or about four-fifths, of the SLA. Any unmeasured overhead or tail variation can consume the remaining 43 ms.

Replace the remote read

Suppose the profile is cached in local memory. The profile access becomes 100 ns, which is 0.0001 ms. Keeping the other two calls sequential gives 0.0001 ms + 5 ms + 2 ms = 7.0001 ms for the named dependencies. Compared with the original 157 ms, the dependency path is reduced by about 157 / 7.0001 ≈ 22 times. The remaining SLA budget is 200 ms − 7.0001 ms = 192.9999 ms, or approximately 193 ms, for application logic, serialization, and variation.

The cache decision can be justified with the same arithmetic. If recomputing the profile costs 10 ms and the object is read 1,000 times per second, recomputation consumes 10 ms × 1,000 = 10,000 ms per second, which is 10 CPU-seconds per second. That is ten seconds of computation demanded every second. An object read once per day does not create the same repeated cost; caching it may add invalidation and consistency complexity without a comparable latency or compute gain.

Parallelize independent work

The critical path changes when calls do not depend on one another. Two independent calls that each take 50 ms cost 50 ms + 50 ms = 100 ms when issued in sequence. Issued in parallel, the endpoint waits approximately for the slower call, so the elapsed time is approximately 50 ms. Parallelism removes about 100 ms − 50 ms = 50 ms, halving this part of the path. It also adds implementation complexity and increases concurrent demand on both services, so use it when the latency budget justifies that cost.

The Price of Optimizing for Latency or Throughput

“Fast” is not a single optimization target. You can reduce the time one request waits, increase the number of requests the system sustains, or try to do both. Those goals often compete: sharing work across records or requests improves utilization, while waiting to accumulate that work and contending for shared resources makes an individual request wait longer.

The comparison is easiest to get wrong for parallel fan-out. Running independent calls concurrently can replace a sum of call latencies with approximately the slowest call, but it does not remove any downstream work. The condition that flips the decision is dependency capacity: fan-out is attractive when the dependencies can absorb the added concurrent load and the coordination and failure handling are acceptable. If a shared dependency is already the constraint, parallelism can move the queue outward without improving sustained throughput.

Batching makes the trade-off especially explicit. Record-at-a-time processing starts work as soon as a record arrives, while micro-batching coalesces records and processes them together. The source describes the reason directly: “processing more records at a time will amortize the overheads associated with processing.” The price is that records wait for the batch, so “the latency per record is higher on average.” The same reasoning applies to higher concurrency: it can raise capacity until queues, memory, or a shared backend become the new ceiling. Optimize for latency when each request has a hard response-time budget; optimize for throughput when sustained volume is the binding constraint. When both matter, keep the critical path short and protect the shared resource rather than declaring the whole system simply fast.

Throughput Ceilings: Find the Bottleneck Before Scaling

Throughput stops scaling when one shared resource becomes the binding constraint. The ceiling may be CPU, memory bandwidth, disk I/O, network I/O, or lock contention. Adding capacity elsewhere does not move that ceiling: faster application code cannot make a saturated disk serve more data, and more application workers cannot make a single database execute more queries.

Use Little’s Law to connect the graphs: average in-flight requests = arrival rate × average latency. At 500 requests/s and 20 ms average latency, the calculation is 500 × 0.020 = 10 requests in flight. That is the relatively healthy baseline. When load rises to 2,000 requests/s against a database that sustains only 500 queries/s, requests accumulate instead of completing. If latency reaches 200 ms, the same calculation becomes 2,000 × 0.200 = 400 requests in flight. Arrival rate has increased fourfold, latency tenfold, and in-flight work fortyfold—from 10 to 400—so queue memory and service failure become part of the failure.

The operator sees throughput flatten near the database’s 500-query/s ceiling while incoming request rate continues upward. Queue depth and in-flight requests climb, latency moves from 20 ms toward 200 ms, and memory usage rises as the service retains waiting work. Adding application instances may spread CPU and network work, but every instance still competes for the same database ceiling. The fleet gets larger without increasing the constrained resource’s service rate.

Diagnose the ceiling before changing the topology. Compare request arrival rate, completed throughput, latency percentiles, queue depth, and in-flight requests. Then correlate them with CPU saturation, memory-bandwidth pressure, disk utilization, network capacity, lock wait time, and database query rate. A flat completed-throughput graph alongside a full database or lock-wait graph identifies a shared constraint; rising CPU with an uncongested database points elsewhere. Request tracing helps locate the slow span when several resources are shared across the path.

The defense must remove work from the constrained resource or distribute that resource’s ownership. For a database bottleneck, cache repeated reads so they do not reach the database, or use read replicas when the workload is read-heavy. If one database writer remains the limit, shard the data so independent keys can be served by separate database partitions. More application instances become useful only after the shared ceiling has moved.

In a Design Conversation: Make the Arithmetic Do the Arguing

A strong design answer starts with a numerical contract, not “this should be fast.” State the SLA and percentile, then show the critical path: which calls are sequential, which fan out in parallel, and what each contributes. If the endpoint allows 10 ms for data access, three database calls at 1 ms each may fit; thirty consume 30 ms before serialization, retries, or application logic. The arithmetic makes the design review concrete.

Volunteer the assumptions

• SLA and percentile: for example, a p99 target rather than an unqualified average.
• Call structure: sequential calls add; independent calls can run in parallel, bounded by the slowest branch.
• Dominant resource: identify whether CPU, disk, network, a database, or a shared lock sets the ceiling.
• Decision ratio: explain why the gap justifies a cache, co-location, replica, or shard.

For a value read 500 times per second, compare the approximately 1 ns cost of an L1-cache hit with the approximately 1 ms cost of a database read. A database read is roughly one million times slower. That ratio is a strong case for an in-process cache if the value tolerates cache invalidation and staleness; the cache is not free, so state those consistency and memory costs instead of presenting it as an automatic win.

When someone claims a throughput number, ask whether it is sustained under steady load or merely a burst peak. When someone quotes latency, ask which percentile accompanies it; an average or p50 can conceal a damaging tail. Then ask what resource was saturated during the measurement. These questions distinguish a measured capacity from a flattering benchmark.

The weak answer is to repeat a latency number without turning it into a constraint: “the database is fast,” “the cache is faster,” or “we will scale horizontally.” Replace each label with an equation, a path, and a bottleneck. Say what budget remains, what can run concurrently, and which shared resource must change when the current ceiling is reached.