Load Balancing: L4 vs. L7, Algorithms, and Health Checks
A practical guide to placing load balancers, choosing between L4 and L7, matching routing algorithms to workload shape, designing dependency-aware health checks, and removing single points of failure in the frontend.
Why One Server Stops Working
A single backend can look sufficient at launch and still fail during the first viral moment. A sudden surge can saturate its CPU, exhaust memory, or consume its network bandwidth; once one resource becomes the bottleneck, every request competes for the same constrained machine. Vertical scaling buys headroom by moving to a larger machine, but that path has a hardware ceiling and a cost cliff. It also leaves the application with one machine to fail, so a hardware or process failure can still take the service offline.
Horizontal scaling removes that concentration by running multiple identical backend servers. The trade-off is a new coordination problem: each incoming request must be assigned to one server, and the assignment must change when a server becomes unavailable. A load balancer sits between clients and the server pool, distributes incoming traffic across the backends, prevents one backend from becoming the bottleneck, and redirects traffic when a backend fails. Azure describes the same mechanism as distributing processing across multiple computing resources to avoid overloading any single resource and improve availability across redundant resources.
Design a platform-scale distributed rate limiter55 min
Critique: find the failure modes in a given design45 min
Horizontal replicas spread the spike and let the survivors absorb traffic when one backend fails.
Trace how horizontal replicas spread incoming traffic and let the remaining backends absorb requests when one backend fails.
L4 and L7 Load Balancers Take Different Paths
The layer determines what the balancer is allowed to know. L4 load balancing distributes connections using transport-level information: the destination IP address and TCP or UDP port. It forwards the flow without reading the HTTP body or interpreting application data. That makes L4 fast and low overhead, but the decision is made once for the connection; the balancer cannot distinguish two requests carried inside it.
L7 load balancing sits at the application layer. It terminates the client’s HTTP or HTTPS connection, parses the request, and can inspect the URL path, headers, cookies, or other request content. It then selects a backend and opens a separate connection to that server. The extra termination, HTTP parsing, and backend connection processing consume CPU and add operational complexity, but they let one entry point route requests by meaning rather than merely by destination.
Dimension
L4
L7
Inspected information
IP addresses and TCP/UDP ports; does not inspect packet data
URLs, HTTP headers, cookies, and other request content
Connection handling
Forwards traffic based on the connection header
Terminates the client HTTP/HTTPS connection and distributes individual requests
Routing decision
Connection-level routing
Content-based routing by URL path, headers, or cookies
Operational cost / security surface
Low overhead; fast because application data inspection is not required
Higher overhead from parsing and inspecting requests; TLS termination adds security responsibilities
The practical boundary is connection-level versus request-level routing. With L4, a long-lived connection remains associated with the backend selected for that flow. With L7, the proxy can make decisions for individual HTTP requests and apply policies such as authentication, rate limiting, logging, or TLS termination. TLS termination means certificates become part of the balancer’s security surface, so certificate rotation and access control belong in its operational design.
That distinction enables content-based routing. /api/* can go to an API pool, /static/* to a static-content origin, and /checkout/* to a PCI-compliant cluster. A ride-sharing service can send WebSocket connections carrying driver-location updates to a dedicated fleet while sending booking REST calls to another pool. An L4 balancer cannot make that choice from payload-blind transport information; it sees the connection, not the URL or application message.
L4 preserves an opaque transport flow; L7 terminates, inspects, and re-establishes the application connection.
Compare where the client connection ends: L4 forwards one opaque TCP flow, while L7 inspects the HTTP request before creating a separate backend connection.
CHECK YOUR UNDERSTANDING
A junior engineer says, “We can use L4 for everything because it is faster.” Describe a case where L4 cannot meet the requirement, and explain what L7 adds.
SHOW ANSWERHIDE ANSWER
If a ride-sharing service must send driver-location WebSocket traffic to one backend pool and booking REST requests to another, L4 cannot distinguish them because it does not inspect application content. L7 terminates the client HTTP/HTTPS connection, reads request details such as the URL or headers, and opens a separate backend connection, making that content-based routing possible.
How the Balancer Chooses a Backend
Once a request reaches the balancer, the selection rule needs a small amount of state: a rotation index, active-connection counts, server weights, or positions on a hash ring. The right choice depends on whether requests cost roughly the same, whether connections remain open, and whether a key must keep returning to the same backend.
Rotation and current load
Round-robin keeps a pointer into the eligible server list. With servers 1, 2, and 3, successive requests go to 1, 2, 3, then wrap back to 1. It equalizes request count, not work. If one request holds a connection while running a slow database query and another finishes quickly, the rotation continues scheduling both as one request each; slow work can accumulate on a server while fast requests arrive behind it. That makes round-robin a good fit for similar servers and similarly costly requests, but a poor fit for variable-cost work.
text
round-robin:
advance rotation to the next eligible server
wrap to the first server after the last
least-outstanding-requests:
choose the server with the fewest active connections
weighted routing:
assign traffic shares proportional to server weights
consistent hashing:
map the key to a point on the ring
route it to the nearest server on that ring
The selectors differ in the state or rule used to choose an eligible backend.
The least-outstanding-requests selector instead examines active connections and sends new work to the server with the fewest. That extra state makes it a better fit for long-lived connections or workloads mixing cheap and expensive queries. It is still a policy you must validate against the workload: a connection count is a proxy for load, not a direct measurement of CPU, memory, or downstream capacity.
Weights and key locality
A weighted variant gives a server a share proportional to its weight. Use higher weights for nodes with greater capacity; during a rolling upgrade, for example, new nodes can receive more traffic while older nodes remain in service. Weighted selection changes the allocation, but it does not make the algorithm load-aware: a node can still receive its assigned share while it is temporarily slow.
Consistent hashing trades broad balancing for stable placement. Hash a client IP, session key, or cache key onto a ring, then route it to the nearest server on that ring. The same key therefore returns to the same backend without storing session state in the balancer. When a node leaves, only a fraction of keys are remapped; the rest retain their placement. That locality is why consistent hashing is useful beyond load balancers, especially for distributed caches where sending a key to a different node can turn a hit into a miss.
CHECK YOUR UNDERSTANDING
A service has requests that take 5ms and others that take 500ms. Why can round-robin cause problems, and which algorithm would you choose first?
SHOW ANSWERHIDE ANSWER
Round-robin assigns equal request counts but ignores how long each request occupies a backend. Several 500ms requests can remain active on one server while the rotation continues sending it more work. Start with least-outstanding-requests, because it considers active work; add weights if the servers have different capacities.
How Health Checks Keep Bad Backends Out
A health check is a periodic probe the load balancer sends to each backend to confirm that the backend can serve traffic. The result changes the backend’s routing state: a healthy server remains eligible, while an unhealthy server is removed from the pool. This decision can be made from real traffic or from synthetic probes, and the distinction determines whether users experience the failure.
Passive versus active detection
A passive check observes production requests. If a real request returns 500 or times out, the balancer records a failure and may mark that backend unhealthy. The mechanism is simple, but the first users routed there have already paid for the failed request. An active check avoids that first-user penalty: at a configured interval, the balancer sends a synthetic request such as GET /healthz. After N consecutive probe failures, it removes the backend before routing more user requests to it. A success resets the failure streak; it does not necessarily re-admit a backend immediately.
Re-admission needs its own threshold. Require M consecutive successes before adding an unhealthy backend back to the pool. Keeping separate failure and success counters prevents one transient success from immediately returning a backend that is still recovering. The exact values of N, M, the probe interval, and the probe timeout are design parameters: lower failure thresholds detect faults sooner, while higher thresholds tolerate transient failures better.
Consecutive probe results determine whether a backend stays eligible, leaves the pool, or returns after recovery.
Follow the probe result into the consecutive-failure or consecutive-success threshold: one branch removes the backend, while the other returns it to the pool.
Make the endpoint test serviceability
A useful /healthz endpoint answers “can this instance serve the traffic the balancer will send?” rather than merely “is the process alive?” Its checks should cover the database, cache, and other required downstream dependencies. A shallow process ping can return success while the database connection pool is exhausted, leaving the instance unable to handle requests. The endpoint should return 503 when a required dependency cannot serve traffic, and 200 only when the required checks pass.
For an e-commerce checkout service, /healthz can perform a lightweight SELECT 1 against the database replica used by the service. If database failover leaves that replica read-only, the check returns 503; the balancer then stops sending write traffic to that node before customers encounter checkout errors. Keep the check lightweight and bounded: a health probe that waits indefinitely on a dependency can consume the same resources needed to recover.
text
GET /healthz
database check:
execute a lightweight SELECT 1 against the service's database replica
dependency checks:
verify required dependencies can serve traffic
if any required check fails:
return 503
otherwise:
return 200
on probe failure:
count consecutive failures
after N consecutive failures, remove backend from pool
on probe success:
count consecutive successes
after M consecutive successes, re-admit backend to pool
The endpoint tests required dependencies, while separate failure and success thresholds control removal and re-admission.
CHECK YOUR UNDERSTANDING
Your health check is an HTTP GET to `/healthz` that returns `200` as long as the process is running. What failure mode does this miss, and how would you improve it?
SHOW ANSWERHIDE ANSWER
It misses an instance that is alive but cannot serve traffic—for example, an exhausted database connection pool, an unreachable cache, or a failed downstream dependency. Make /healthz verify the dependencies required for real requests, such as executing a lightweight SELECT 1, and return 503 when a required check fails. Use consecutive failure and success thresholds so the balancer removes a failing instance and re-admits it only after it has recovered.
Worked Example: Surviving a 10x Food-Delivery Spike
Capacity arithmetic
On launch day, a food-delivery app receives 10× its normal order traffic. Its original backend API server reaches 100% CPU, so adding more work to that server is no longer a scaling strategy. Run three identical API servers behind a load balancer instead. The incoming work is divided into three shares: 10 ÷ 3 = 3.33, so each server receives roughly one-third of the traffic, or 33% of the requests. The 33% figure is each server’s share of incoming work; it is not a claim that CPU utilization will also be exactly 33%.
The arrangement also changes the failure calculation. If one server falls over and health checks remove it from rotation, the same launch traffic still has to be served by the two survivors: 10 ÷ 2 = 5 units of work per server. Before the failure, each survivor handled 10 ÷ 3 = 3.33 units. Its new share is therefore 5 ÷ 3.33 ≈ 1.5× the old share: each remaining server absorbs the failed server’s one-third share and now handles roughly 50% of the traffic. Whether that remains within capacity is the sizing decision; the load balancer provides redistribution, not extra compute.
Session state determines whether those choices remain interchangeable. With stateless API servers, session data lives outside the process, so any healthy server can handle the next request and you can choose round-robin or least-connections based on request cost. With stateful sessions, sending a user to a different server can lose in-memory state; use session affinity or consistent hashing so related requests keep reaching the required backend. That preserves continuity, but couples routing to server membership and failure behavior.
CHECK YOUR UNDERSTANDING
After one of the three servers fails during the 10× spike, what fraction of the traffic does each survivor receive, and how does that compare with its pre-failure share?
SHOW ANSWERHIDE ANSWER
Each survivor receives 10 ÷ 2 = 5 units, or roughly 50% of the traffic. Before the failure it received 10 ÷ 3 = 3.33 units, or roughly 33%. Its share increases by 5 ÷ 3.33 ≈ 1.5×.
Choosing the Layer, Algorithm, and Session Model
Choose the layer and algorithm from the work the backend must preserve, not from a blanket preference for speed. An external L7 load balancer or API gateway is a useful internet edge: it can inspect HTTP requests, terminate TLS, and route by paths, headers, methods, or cookies. The price is an extra network hop plus CPU and memory for TLS termination, HTTP parsing, and policy management. Between internal service tiers, L4 is often the better fit when the protocol is simple and you want lower processing overhead; it routes connections using transport-level information without understanding application content.
Compare the layer and algorithm choices by what they inspect, where they route, and the workload shape they fit.
Choice
Inspects
Routing basis
Best-fit traffic shape
Principal cost or coupling
L4
IP addresses, TCP, or UDP ports
Connections
Simple TCP/UDP workloads
Low overhead; blind to application content
L7
HTTP headers, URLs, cookies, and other request content
Paths, headers, methods, or cookies
HTTP(S), HTTP/2, gRPC, and routing based on request attributes such as paths, headers, or cookies
Parsing and inspection cost CPU and memory; TLS termination and policy management
Round-robin
No current server state
Requests cycle through servers in turn
Similar request costs and homogeneous capacity
Equal request counts can still produce uneven load when request costs vary
Least-connections
Current active connections
Server with the fewest active connections
Variable request processing times and long-lived connections
Requires tracking connection counts; connections are only a proxy for load
Weighted routing
Configured server weights or capacities
Proportionally more traffic to higher-capacity servers
Heterogeneous capacity, including rolling upgrades
Requires weight assignment and tuning
Consistent hashing
Client IP or session key
Nearest server on a hash ring
Session or cache-key locality with stateful backends
Adding or removing a node remaps only a fraction of keys, but introduces backend coupling
The algorithm decision follows the traffic shape. Use round-robin when requests have roughly similar costs and servers have comparable capacity. If request duration varies or connections are long-lived, least-connections is a better starting point, although connection count remains only a proxy for actual load. Weighted routing handles heterogeneous capacity: during a rolling upgrade, for example, a higher-capacity server can receive a larger share while older servers remain in the pool. Consistent hashing is the choice when locality matters—such as mapping a session key or cache key to the same backend—because membership changes remap only a fraction of keys rather than reshuffling every key.
The decision flips when application visibility or state is non-negotiable. L4 is the right tool for simple TCP/UDP traffic where content-based routing is unnecessary; it cannot distinguish requests inside a multiplexed application connection. L7 earns its cost when you need HTTP-aware routing, security policy, or protocol awareness—for example, separating API paths or routing HTTP/2 and gRPC requests by application attributes. In a stateful WebSocket service, round-robin can send successive connections to different servers without preserving the state each connection needs; affinity or consistent hashing keeps the connection's identity near its state. In an interview, name the layer at the edge and between tiers, then tie the algorithm to request variance, capacity, connection lifetime, and locality.
Failure Mode: Shallow Checks and Flapping Backends
A shallow GET /healthz can return 200 while the instance is unable to serve real traffic. The process may be alive even though its database connection pool, cache, downstream dependency, or thread pool is exhausted. Passive checking discovers that only after a real request fails or times out, so the first users pay the failure cost before the balancer removes the instance.
Active checks move detection earlier, but their timeout is a stability trade-off. A probe timeout that is shorter than a normal load spike produces false failures; a timeout that is too long leaves an unhealthy instance serving traffic while probes hang. Azure's health-check guidance gives the relationship as App < Probe < Period x Threshold: the application should time out before the probe, and the probe should finish before the failure window closes.
Flapping makes the problem worse. Without separate failure and success thresholds, a backend can be removed and re-admitted too readily as probe results alternate. The backend repeatedly enters and leaves rotation, while traffic and connection load shift among the remaining servers. Use hysteresis: require a failure threshold before removal and M consecutive successes before re-admission. In one example readiness configuration, three failures at five-second intervals produce a 15-second removal window, while two successes at five-second intervals produce a 10-second recovery window; use such thresholds as design parameters rather than universal defaults. That trades slower recovery for a less trigger-happy pool.
The checkout case shows why dependency-aware readiness matters. If /healthz performs a lightweight SELECT 1 against the database replica and the replica becomes read-only during failover, the endpoint returns 503. The balancer can then stop sending write traffic to that node before customers encounter checkout errors. Keep dependency probes bounded and lightweight: checking every downstream system can itself consume the resources the instance needs to recover.
A failed probe removes the backend, but a single recovery probe does not immediately put it back into rotation.
Follow the path from the first failed probe through removal, then note that one success is insufficient and consecutive successes are required before re-admission.
CHECK YOUR UNDERSTANDING
A readiness probe returns 200 whenever the process is alive, but checkout errors rise during a database failover. What is the gap, and what change prevents the failed node from receiving traffic?
SHOW ANSWERHIDE ANSWER
The probe checks liveness rather than the dependency required by checkout. Make readiness dependency-aware—for example, verify the database path and return 503 when the replica is read-only—then require consecutive successful probes before re-admission.
Failure Mode: The Load Balancer Becomes the Outage
A load balancer can become the outage it was supposed to prevent. If the only frontend process fails, clients can no longer reach any healthy backend—even when every backend is still serving traffic. The failure sequence is straightforward: the frontend address points at the failed balancer, connection attempts stop there, and the backend fleet receives no new work. Operators see frontend connection failures while backend CPU and request volume fall, which can misleadingly make the backend tier look healthy.
Remove that single failure domain by running a redundant pair. In an active-passive arrangement, one balancer owns the shared virtual IP (VIP) and forwards traffic; its peer monitors it and takes ownership after failure. In active-active, both balancers serve traffic, so losing one reduces frontend capacity rather than removing the path entirely. VRRP keeps the VIP associated with the current owner during failover, allowing clients to continue using one frontend address instead of learning a new address.
A practical topology puts an external L7 load balancer—or an API gateway that performs that role—at the internet edge. It can route HTTP traffic to the appropriate service tier. Internal L4 balancers then distribute transport traffic between service tiers when lower-latency routing is more important than application-aware routing. Azure's guidance notes that an effective setup often uses more than one type of load-balancing solution at different places in the workload; Google Cloud documents external load balancing for internet clients and internal load balancing for clients inside the cloud.
Redundancy must also include backend membership. Each balancer should apply the health-check policy to its own pool and remove a backend when the check says it cannot serve traffic. Otherwise, a partial backend failure can continue returning errors through both frontend nodes and become a full service outage. The pair protects the entry point; health-based removal protects the destinations.
A shared VIP survives loss of one external balancer while backend health checks keep failed destinations out of rotation.
Follow the single VIP from internet clients through the redundant external pair, then compare the active-active and active-passive ownership paths before tracing traffic through the internal tier to healthy backends.
CHECK YOUR UNDERSTANDING
Your load balancer is itself the single point of failure. Which two deployment patterns remove that risk, and what mechanism keeps the virtual IP available during failover?
SHOW ANSWERHIDE ANSWER
Run the balancers as an active-active pair or an active-passive pair. Use a shared virtual IP and VRRP so the surviving or active node takes ownership of the frontend address when its peer fails; clients keep using the same address.
How to Defend the Design in an Interview
A strong design answer names the load balancer’s position before naming the product. For internet traffic, place an external L7 balancer or API gateway at the edge when you need HTTP routing, TLS termination, or session-aware behavior. Between internal service tiers, an L4 balancer may be the better fit when routing by IP and TCP or UDP port is sufficient and you want lower overhead. State the boundary explicitly: “The edge balancer routes HTTP requests by path; internal traffic uses L4 because the services do not need payload-aware routing.”
Then connect the algorithm to the traffic pattern. Use round-robin for similarly sized, short-lived requests. Use least-connections or least-outstanding-requests when request cost varies or connections are long-lived. Use weights when backend capacity differs, such as during a rolling upgrade. For a distributed cache or another stateful partitioned workload, use consistent hashing so the same key reaches the same node and membership changes remap only a fraction of keys.
Health checks should answer “can this backend serve this class of traffic?”, not merely “is the process running?” Describe the probe, its interval and failure threshold, then name the dependencies it verifies: for example, database connectivity, cache reachability, or a required downstream service. Also state the recovery rule, such as requiring consecutive successful probes before re-admission, so a flapping backend does not thrash in and out of rotation.
Finally, explain the session model. Keep application backends stateless where possible and put shared session state in a shared cache such as Redis; then any healthy backend can serve a request. A stateful WebSocket service is different: round-robin does not provide the affinity needed to keep successive WebSocket connections or related state on the required backend. Use consistent hashing or cookie-based affinity to keep a client on the backend holding its connection state. For availability, run the balancer as an active-active or active-passive pair and preserve the shared virtual IP with VRRP.
CHECK YOUR UNDERSTANDING
A stateful WebSocket service has long-lived connections. What should your answer say about routing and why?
SHOW ANSWERHIDE ANSWER
Round-robin distributes new connections but does not provide affinity for subsequent connections from the same client. Use consistent hashing or cookie-based affinity so later connections stay with the backend holding the relevant connection state.
KEY TAKEAWAYS
L4 routes opaque transport connections, while L7 terminates and inspects application requests for content-based routing, TLS termination, and policy enforcement.
Round-robin equalizes request count; least-outstanding-requests responds to active work; weights handle heterogeneous capacity; consistent hashing preserves key locality as membership changes.
Health checks must test whether an instance can serve real traffic, not merely whether its process is alive, and should use separate failure and recovery thresholds.
A load balancer needs its own redundancy through active-active or active-passive deployment and a shared VIP, while health checks protect the backend pool.