Latency, Throughput, and the Numbers That Shape System Design
Learn how to turn latency and throughput figures into design constraints: distinguish per-request response time from sustained capacity, use order-of-magnitude latency estimates, budget sequential and parallel work, identify shared-resource bottlenecks, and make performance trade-offs explicit in a design review.
“Fast” Describes Two Different Things
“Fast” is ambiguous in a design review. A request can finish quickly while the system handles only a small number of requests, or the system can complete many requests per second while each request waits in a queue. If you optimize only the first property, you may leave capacity unused; if you optimize only the second, you can make individual users wait longer.
Consider a single-threaded HTTP server handling 1,000 requests per second at 5 ms p99. Switching to 100 threads can raise throughput to 20,000 requests per second—20 times as much—but p99 can reach 50 ms, 10 times higher, because the threads contend for the same database connection pool. The system is better at absorbing work, but worse at completing one request promptly.
Design a platform-scale distributed rate limiter55 min
Critique: find the failure modes in a given design45 min
A short individual response and high completed volume are different properties of a system.
Compare the left path's short per-request journey with the right path's larger queue and higher completed volume.
The distinction changes what you measure and what you optimize. A user-facing endpoint usually needs a bounded response time; a batch or ingestion pipeline may need sustained capacity. Treating a capacity problem as a latency problem leads you to shave time from individual operations without removing the shared bottleneck. Treating a latency problem as a throughput problem leads you to add concurrency and may worsen the queue that users are already waiting in.
Latency and throughput are therefore separate design dimensions that interact under load, not two names for the same notion of speed. The next mechanism gives each dimension its own measurement so you can budget response time without losing sight of system capacity.
Latency: One Operation, One Clock
Latency is the elapsed time for one operation, measured from the moment the request bytes leave the sender until the response is fully received. That clock includes every hop on the critical path: transmission, propagation, server-side queuing, computation, I/O, and the return path. A request is not complete when the server begins processing it; it is complete when the caller has the response.
Separate the clock into two useful categories. Propagation latency is the time required for signals to travel through physical distance. It is constrained by the speed of light, so moving a dependency closer is the architectural lever; tuning application code cannot remove that distance. Processing latency is everything your system adds while handling the request: waiting in a queue, running computation, reading storage, or serializing the response. A US-East to US-West round trip has approximately 60 ms of propagation latency imposed by physics alone. Every millisecond above that propagation floor comes from additional processing, queuing, transmission, or network-path latency.
Latency is the complete elapsed time: propagation and processing must finish before the response is received, while percentiles expose the slow tail.
Follow the request from the sender through propagation and processing to the fully received response; the tail marker shows why the slowest requests need their own budget.
In production, report latency as a percentile rather than only an average. p50 is the median request: half complete faster and half slower. p99 is the point below which 99% of requests complete, leaving 1 in 100 beyond it. p999 exposes an even deeper tail. The average can look healthy while a small group waits far longer, because fast requests pull the mean down. For an SLA, that tail is not an outlier you can ignore; it is the experience of a predictable fraction of users.
Use the split to diagnose the right class of problem. A higher propagation component points to distance or the number of network crossings. A higher processing component points to queueing, compute, I/O, or work added inside the request path. The same endpoint can therefore have a good median and a bad tail when contention occasionally makes processing wait, while a remote dependency can impose a floor that no amount of local optimization can cross.
CHECK YOUR UNDERSTANDING
A teammate says, “Our average response time is 8 ms, so we’re in great shape.” What do you ask before accepting that claim?
SHOW ANSWERHIDE ANSWER
Ask for the p50, p99, and p999 latencies, not just the average, and verify the measurement boundaries: does the clock include network time and the fully received response? Check the workload and time window, then look for tail causes such as queueing, I/O, or a slow downstream hop. An 8 ms average is insufficient if the p99 violates the endpoint’s SLA.
Throughput: Sustained Capacity, Not a Burst Peak
Throughput is the number of operations a system completes per unit of time: requests per second, records per second, or bytes per second. Measure it under a sustained, steady load, not from the highest short burst the system can absorb. A burst peak tells you how much work fits temporarily in buffers; sustained throughput tells you how much work the system can keep completing without the queue growing.
Trace each request through the resources it consumes. CPU work, network transfer, and disk I/O may all contribute, but the slowest shared resource establishes the ceiling. If a disk serves 500 MB/s and each query scans 5 MB, the disk can support 100 QPS: 500 divided by 5. A faster CPU cannot raise that limit because every additional query still needs the same disk service.
The calculation is deliberately simple: name the resource's service rate, name the work required by one operation, and divide the first by the second.
The result is 100 QPS, not 500 QPS: each query consumes 5 MB of the disk's 500 MB/s budget. That ceiling applies before application overhead, contention, or other disk users consume capacity, so it is an upper bound rather than a performance promise.
Requests pass through upstream stages before waiting for the shared disk, whose service rate limits completed operations.
Follow the request path into the shared disk queue: faster CPU or network stages cannot make the disk complete more than its service capacity.
Concurrency raises capacity—and queues
Adding workers can increase throughput when the original worker was idle or was the limiting resource. It can also increase per-request latency: more requests reach the same database connection pool or disk, compete for service, and wait in a longer queue. The system may complete more work per second while each individual request takes longer.
A concrete case makes the trade-off visible. A single-threaded HTTP server handles 1,000 requests/s at 5 ms p99. Moving to 100 threads raises throughput to 20,000 requests/s—a 20× increase—but p99 rises to 50 ms, 10× higher, because the threads contend for the same database connection pool. The extra threads improved concurrency, not the pool's underlying service capacity.
CHECK YOUR UNDERSTANDING
A system's throughput rises after you add concurrency, but p99 latency rises sharply too. What changed?
SHOW ANSWERHIDE ANSWER
More requests are being processed concurrently, but they are sharing a constrained resource. The added concurrency raises completed work until that resource saturates; the resulting queue increases per-request latency, so throughput and latency move in opposite directions.
Jeff Dean’s Latency Numbers: A Ruler for Architecture
Jeff Dean’s latency numbers are a cost ladder, not a set of trivia questions. They give you an order-of-magnitude estimate before you have a benchmark: an L1 cache reference is about 1 ns, main memory about 100 ns, a random SSD read about 100 µs, a spinning-disk seek about 10 ms, and a cross-datacenter round trip about 150 ms. The values vary with hardware and workload; use them as a ruler for architecture, then replace them with measurements for your system.
Each step outward makes an operation increasingly expensive relative to the previous level.
Read the ladder as multiplicative gaps: moving one level outward changes the budget by orders of magnitude, not by a small constant.
The ratios are the useful part. Main memory is roughly 1,000× faster than a random SSD read, and an SSD read is roughly 100× faster than a spinning-disk seek. A same-datacenter round trip of about 500 µs is roughly 500,000 1-ns L1-cache references. That is why replacing one remote hop with a local lookup can matter more than shaving a few instructions from application code. The exact ratio depends on which cache level, storage operation, and network path you compare.
Use the ladder by translating each operation into the budget it consumes. Ten sequential SSD reads at roughly 100 µs each cost about 1 ms; ten disk seeks at roughly 10 ms each cost about 100 ms. Conversely, one cross-datacenter round trip can consume an entire tight request budget. If your SLA is 100 ms at p99, a required cross-datacenter call costing about 150 ms has already exceeded it before application logic, serialization, or data access runs. Put that data closer to the caller or serve it from a local cache instead of trying to optimize code after the remote call.
CHECK YOUR UNDERSTANDING
A required downstream service lives in another datacenter, and your endpoint has a 150 ms SLA. Is the design viable, and what would you change?
SHOW ANSWERHIDE ANSWER
Not if the cross-datacenter round trip is about 150 ms: it consumes the entire SLA before your application does any work, leaving no budget for serialization, processing, or other dependencies. Co-locate the required data or service with the caller, or cache the value locally so the request does not require that remote round trip.
Worked Example: Budget a Feed Endpoint Before You Build It
Start with the SLA
Treat the 200 ms p99 SLA as a budget, not as a number you check after implementation. Every dependency on the request's critical path spends part of that budget. The feed-render endpoint needs three results: a user profile from another datacenter, ranked posts from a local SSD, and advertisements from a service in the same datacenter. Their stated costs are approximately 150 ms, 5 ms, and 2 ms.
If the endpoint calls them sequentially, the arithmetic is direct: 150 ms + 5 ms + 2 ms = 157 ms. That leaves 200 ms − 157 ms = 43 ms for request handling, serialization, and any other work. The design fits on paper, but the margin is small: the known dependency cost consumes 157 / 200 = 78.5%, or about four-fifths, of the SLA. Any unmeasured overhead or tail variation can consume the remaining 43 ms.
Replace the remote read
Suppose the profile is cached in local memory. The profile access becomes 100 ns, which is 0.0001 ms. Keeping the other two calls sequential gives 0.0001 ms + 5 ms + 2 ms = 7.0001 ms for the named dependencies. Compared with the original 157 ms, the dependency path is reduced by about 157 / 7.0001 ≈ 22 times. The remaining SLA budget is 200 ms − 7.0001 ms = 192.9999 ms, or approximately 193 ms, for application logic, serialization, and variation.
The cache decision can be justified with the same arithmetic. If recomputing the profile costs 10 ms and the object is read 1,000 times per second, recomputation consumes 10 ms × 1,000 = 10,000 ms per second, which is 10 CPU-seconds per second. That is ten seconds of computation demanded every second. An object read once per day does not create the same repeated cost; caching it may add invalidation and consistency complexity without a comparable latency or compute gain.
Parallelize independent work
The critical path changes when calls do not depend on one another. Two independent calls that each take 50 ms cost 50 ms + 50 ms = 100 ms when issued in sequence. Issued in parallel, the endpoint waits approximately for the slower call, so the elapsed time is approximately 50 ms. Parallelism removes about 100 ms − 50 ms = 50 ms, halving this part of the path. It also adds implementation complexity and increases concurrent demand on both services, so use it when the latency budget justifies that cost.
CHECK YOUR UNDERSTANDING
You need data from three independent services to render a page, and each call takes approximately 40 ms. How should you implement the calls, and what trade-off does that introduce?
SHOW ANSWERHIDE ANSWER
Sequential calls take 40 ms + 40 ms + 40 ms = 120 ms. Issuing the three independent calls in parallel makes the ideal elapsed time approximately 40 ms, a reduction of 120 ms − 40 ms = 80 ms, or about three times less latency for this portion of the request. The trade-off is added concurrency and coordination: all three calls now run at once, and the endpoint still depends on the slowest result and must handle failures or timeouts from each service.
The Price of Optimizing for Latency or Throughput
“Fast” is not a single optimization target. You can reduce the time one request waits, increase the number of requests the system sustains, or try to do both. Those goals often compete: sharing work across records or requests improves utilization, while waiting to accumulate that work and contending for shared resources makes an individual request wait longer.
Compare each move by the constraint it improves and the resource or complexity it adds.
Design move
Primary benefit
Latency effect
Throughput effect
Price or new bottleneck
Local cache
Keep frequently accessed data closer
Reduces the delay of repeated reads
Reduces backend work for cacheable reads
Consumes memory; cache invalidation and stale data add complexity
Parallel fan-out
Overlap independent downstream work
Replaces a sum of call latencies with approximately the slowest call
Does not remove downstream work; increases concurrent load on dependencies
Requires coordination and failure handling across dependencies
Batching or higher concurrency
Amortize per-record overhead or process more records at once
Batch waiting and resource contention increase per-record latency
Raises throughput by amortizing processing overhead
Queues, shared-resource contention, and memory usage can become the bottleneck
Horizontal application scaling
Add application capacity across instances
May reduce application-side queuing
Raises capacity when the application tier is the constraint
A single shared database, global lock, or network link remains the bottleneck
The comparison is easiest to get wrong for parallel fan-out. Running independent calls concurrently can replace a sum of call latencies with approximately the slowest call, but it does not remove any downstream work. The condition that flips the decision is dependency capacity: fan-out is attractive when the dependencies can absorb the added concurrent load and the coordination and failure handling are acceptable. If a shared dependency is already the constraint, parallelism can move the queue outward without improving sustained throughput.
Batching makes the trade-off especially explicit. Record-at-a-time processing starts work as soon as a record arrives, while micro-batching coalesces records and processes them together. The source describes the reason directly: “processing more records at a time will amortize the overheads associated with processing.” The price is that records wait for the batch, so “the latency per record is higher on average.” The same reasoning applies to higher concurrency: it can raise capacity until queues, memory, or a shared backend become the new ceiling. Optimize for latency when each request has a hard response-time budget; optimize for throughput when sustained volume is the binding constraint. When both matter, keep the critical path short and protect the shared resource rather than declaring the whole system simply fast.
Throughput Ceilings: Find the Bottleneck Before Scaling
Throughput stops scaling when one shared resource becomes the binding constraint. The ceiling may be CPU, memory bandwidth, disk I/O, network I/O, or lock contention. Adding capacity elsewhere does not move that ceiling: faster application code cannot make a saturated disk serve more data, and more application workers cannot make a single database execute more queries.
Use Little’s Law to connect the graphs: average in-flight requests = arrival rate × average latency. At 500 requests/s and 20 ms average latency, the calculation is 500 × 0.020 = 10 requests in flight. That is the relatively healthy baseline. When load rises to 2,000 requests/s against a database that sustains only 500 queries/s, requests accumulate instead of completing. If latency reaches 200 ms, the same calculation becomes 2,000 × 0.200 = 400 requests in flight. Arrival rate has increased fourfold, latency tenfold, and in-flight work fortyfold—from 10 to 400—so queue memory and service failure become part of the failure.
A database ceiling turns excess arrival rate into queue growth, rising latency, and memory exhaustion.
Trace the overload from the 2,000 requests/s arrival rate through the 500-queries/s database ceiling to the queue, 200 ms latency, 400 in-flight requests, and exhausted memory.
The operator sees throughput flatten near the database’s 500-query/s ceiling while incoming request rate continues upward. Queue depth and in-flight requests climb, latency moves from 20 ms toward 200 ms, and memory usage rises as the service retains waiting work. Adding application instances may spread CPU and network work, but every instance still competes for the same database ceiling. The fleet gets larger without increasing the constrained resource’s service rate.
Diagnose the ceiling before changing the topology. Compare request arrival rate, completed throughput, latency percentiles, queue depth, and in-flight requests. Then correlate them with CPU saturation, memory-bandwidth pressure, disk utilization, network capacity, lock wait time, and database query rate. A flat completed-throughput graph alongside a full database or lock-wait graph identifies a shared constraint; rising CPU with an uncongested database points elsewhere. Request tracing helps locate the slow span when several resources are shared across the path.
The defense must remove work from the constrained resource or distribute that resource’s ownership. For a database bottleneck, cache repeated reads so they do not reach the database, or use read replicas when the workload is read-heavy. If one database writer remains the limit, shard the data so independent keys can be served by separate database partitions. More application instances become useful only after the shared ceiling has moved.
CHECK YOUR UNDERSTANDING
Your service stops scaling after you add the fifth application server. What are three possible reasons, and how would you diagnose which one it is?
SHOW ANSWERHIDE ANSWER
A shared database may be capped, a global lock may serialize work, or another resource such as CPU, disk I/O, memory bandwidth, or network I/O may be saturated. Compare completed throughput with arrival rate, then correlate latency and queue depth with database query rate, lock waits, CPU, memory, disk, and network metrics. Traces can identify the slow span. If the database is the ceiling, use read replicas or a cache; if data ownership is concentrated, shard it; if CPU is saturated, application scaling may help.
In a Design Conversation: Make the Arithmetic Do the Arguing
A strong design answer starts with a numerical contract, not “this should be fast.” State the SLA and percentile, then show the critical path: which calls are sequential, which fan out in parallel, and what each contributes. If the endpoint allows 10 ms for data access, three database calls at 1 ms each may fit; thirty consume 30 ms before serialization, retries, or application logic. The arithmetic makes the design review concrete.
Volunteer the assumptions
SLA and percentile: for example, a p99 target rather than an unqualified average.
Call structure: sequential calls add; independent calls can run in parallel, bounded by the slowest branch.
Dominant resource: identify whether CPU, disk, network, a database, or a shared lock sets the ceiling.
Decision ratio: explain why the gap justifies a cache, co-location, replica, or shard.
For a value read 500 times per second, compare the approximately 1 ns cost of an L1-cache hit with the approximately 1 ms cost of a database read. A database read is roughly one million times slower. That ratio is a strong case for an in-process cache if the value tolerates cache invalidation and staleness; the cache is not free, so state those consistency and memory costs instead of presenting it as an automatic win.
When someone claims a throughput number, ask whether it is sustained under steady load or merely a burst peak. When someone quotes latency, ask which percentile accompanies it; an average or p50 can conceal a damaging tail. Then ask what resource was saturated during the measurement. These questions distinguish a measured capacity from a flattering benchmark.
The weak answer is to repeat a latency number without turning it into a constraint: “the database is fast,” “the cache is faster,” or “we will scale horizontally.” Replace each label with an equation, a path, and a bottleneck. Say what budget remains, what can run concurrently, and which shared resource must change when the current ceiling is reached.
KEY TAKEAWAYS
Latency measures how long one operation takes; throughput measures how much work completes over time, and improving one can worsen the other.
Use p50, p99, and p999 rather than averages alone so tail latency is visible in an SLA discussion.
Budget sequential work by adding its costs, while independent parallel work is bounded approximately by the slowest branch.
A shared resource such as a disk, database, connection pool, or lock sets the throughput ceiling when every request still depends on it.
Turn performance claims into arithmetic: compare operation costs with the SLA and scale the resource that actually limits throughput.