Load Balancing: L4 vs. L7, Algorithms, and Health Checks

Why One Server Stops Working

A single backend can look sufficient at launch and still fail during the first viral moment. A sudden surge can saturate its CPU, exhaust memory, or consume its network bandwidth; once one resource becomes the bottleneck, every request competes for the same constrained machine. Vertical scaling buys headroom by moving to a larger machine, but that path has a hardware ceiling and a cost cliff. It also leaves the application with one machine to fail, so a hardware or process failure can still take the service offline.

Horizontal scaling removes that concentration by running multiple identical backend servers. The trade-off is a new coordination problem: each incoming request must be assigned to one server, and the assignment must change when a server becomes unavailable. A load balancer sits between clients and the server pool, distributes incoming traffic across the backends, prevents one backend from becoming the bottleneck, and redirects traffic when a backend fails. Azure describes the same mechanism as distributing processing across multiple computing resources to avoid overloading any single resource and improve availability across redundant resources.

L4 and L7 Load Balancers Take Different Paths

The layer determines what the balancer is allowed to know. L4 load balancing distributes connections using transport-level information: the destination IP address and TCP or UDP port. It forwards the flow without reading the HTTP body or interpreting application data. That makes L4 fast and low overhead, but the decision is made once for the connection; the balancer cannot distinguish two requests carried inside it.

L7 load balancing sits at the application layer. It terminates the client’s HTTP or HTTPS connection, parses the request, and can inspect the URL path, headers, cookies, or other request content. It then selects a backend and opens a separate connection to that server. The extra termination, HTTP parsing, and backend connection processing consume CPU and add operational complexity, but they let one entry point route requests by meaning rather than merely by destination.

The practical boundary is connection-level versus request-level routing. With L4, a long-lived connection remains associated with the backend selected for that flow. With L7, the proxy can make decisions for individual HTTP requests and apply policies such as authentication, rate limiting, logging, or TLS termination. TLS termination means certificates become part of the balancer’s security surface, so certificate rotation and access control belong in its operational design.

That distinction enables content-based routing. /api/* can go to an API pool, /static/* to a static-content origin, and /checkout/* to a PCI-compliant cluster. A ride-sharing service can send WebSocket connections carrying driver-location updates to a dedicated fleet while sending booking REST calls to another pool. An L4 balancer cannot make that choice from payload-blind transport information; it sees the connection, not the URL or application message.

How the Balancer Chooses a Backend

Once a request reaches the balancer, the selection rule needs a small amount of state: a rotation index, active-connection counts, server weights, or positions on a hash ring. The right choice depends on whether requests cost roughly the same, whether connections remain open, and whether a key must keep returning to the same backend.

Rotation and current load

Round-robin keeps a pointer into the eligible server list. With servers 1, 2, and 3, successive requests go to 1, 2, 3, then wrap back to 1. It equalizes request count, not work. If one request holds a connection while running a slow database query and another finishes quickly, the rotation continues scheduling both as one request each; slow work can accumulate on a server while fast requests arrive behind it. That makes round-robin a good fit for similar servers and similarly costly requests, but a poor fit for variable-cost work.

round-robin:
    advance rotation to the next eligible server
    wrap to the first server after the last

least-outstanding-requests:
    choose the server with the fewest active connections

weighted routing:
    assign traffic shares proportional to server weights

consistent hashing:
    map the key to a point on the ring
    route it to the nearest server on that ring

The least-outstanding-requests selector instead examines active connections and sends new work to the server with the fewest. That extra state makes it a better fit for long-lived connections or workloads mixing cheap and expensive queries. It is still a policy you must validate against the workload: a connection count is a proxy for load, not a direct measurement of CPU, memory, or downstream capacity.

Weights and key locality

A weighted variant gives a server a share proportional to its weight. Use higher weights for nodes with greater capacity; during a rolling upgrade, for example, new nodes can receive more traffic while older nodes remain in service. Weighted selection changes the allocation, but it does not make the algorithm load-aware: a node can still receive its assigned share while it is temporarily slow.

Consistent hashing trades broad balancing for stable placement. Hash a client IP, session key, or cache key onto a ring, then route it to the nearest server on that ring. The same key therefore returns to the same backend without storing session state in the balancer. When a node leaves, only a fraction of keys are remapped; the rest retain their placement. That locality is why consistent hashing is useful beyond load balancers, especially for distributed caches where sending a key to a different node can turn a hit into a miss.

How Health Checks Keep Bad Backends Out

A health check is a periodic probe the load balancer sends to each backend to confirm that the backend can serve traffic. The result changes the backend’s routing state: a healthy server remains eligible, while an unhealthy server is removed from the pool. This decision can be made from real traffic or from synthetic probes, and the distinction determines whether users experience the failure.

Passive versus active detection

A passive check observes production requests. If a real request returns 500 or times out, the balancer records a failure and may mark that backend unhealthy. The mechanism is simple, but the first users routed there have already paid for the failed request. An active check avoids that first-user penalty: at a configured interval, the balancer sends a synthetic request such as GET /healthz. After N consecutive probe failures, it removes the backend before routing more user requests to it. A success resets the failure streak; it does not necessarily re-admit a backend immediately.

Re-admission needs its own threshold. Require M consecutive successes before adding an unhealthy backend back to the pool. Keeping separate failure and success counters prevents one transient success from immediately returning a backend that is still recovering. The exact values of N, M, the probe interval, and the probe timeout are design parameters: lower failure thresholds detect faults sooner, while higher thresholds tolerate transient failures better.

Make the endpoint test serviceability

A useful /healthz endpoint answers “can this instance serve the traffic the balancer will send?” rather than merely “is the process alive?” Its checks should cover the database, cache, and other required downstream dependencies. A shallow process ping can return success while the database connection pool is exhausted, leaving the instance unable to handle requests. The endpoint should return 503 when a required dependency cannot serve traffic, and 200 only when the required checks pass.

For an e-commerce checkout service, /healthz can perform a lightweight SELECT 1 against the database replica used by the service. If database failover leaves that replica read-only, the check returns 503; the balancer then stops sending write traffic to that node before customers encounter checkout errors. Keep the check lightweight and bounded: a health probe that waits indefinitely on a dependency can consume the same resources needed to recover.

GET /healthz

database check:
    execute a lightweight SELECT 1 against the service's database replica
dependency checks:
    verify required dependencies can serve traffic

if any required check fails:
    return 503
otherwise:
    return 200

on probe failure:
    count consecutive failures
    after N consecutive failures, remove backend from pool

on probe success:
    count consecutive successes
    after M consecutive successes, re-admit backend to pool

Worked Example: Surviving a 10x Food-Delivery Spike

Capacity arithmetic

On launch day, a food-delivery app receives 10× its normal order traffic. Its original backend API server reaches 100% CPU, so adding more work to that server is no longer a scaling strategy. Run three identical API servers behind a load balancer instead. The incoming work is divided into three shares: 10 ÷ 3 = 3.33, so each server receives roughly one-third of the traffic, or 33% of the requests. The 33% figure is each server’s share of incoming work; it is not a claim that CPU utilization will also be exactly 33%.

The arrangement also changes the failure calculation. If one server falls over and health checks remove it from rotation, the same launch traffic still has to be served by the two survivors: 10 ÷ 2 = 5 units of work per server. Before the failure, each survivor handled 10 ÷ 3 = 3.33 units. Its new share is therefore 5 ÷ 3.33 ≈ 1.5× the old share: each remaining server absorbs the failed server’s one-third share and now handles roughly 50% of the traffic. Whether that remains within capacity is the sizing decision; the load balancer provides redistribution, not extra compute.

Session state determines whether those choices remain interchangeable. With stateless API servers, session data lives outside the process, so any healthy server can handle the next request and you can choose round-robin or least-connections based on request cost. With stateful sessions, sending a user to a different server can lose in-memory state; use session affinity or consistent hashing so related requests keep reaching the required backend. That preserves continuity, but couples routing to server membership and failure behavior.

Choosing the Layer, Algorithm, and Session Model

Choose the layer and algorithm from the work the backend must preserve, not from a blanket preference for speed. An external L7 load balancer or API gateway is a useful internet edge: it can inspect HTTP requests, terminate TLS, and route by paths, headers, methods, or cookies. The price is an extra network hop plus CPU and memory for TLS termination, HTTP parsing, and policy management. Between internal service tiers, L4 is often the better fit when the protocol is simple and you want lower processing overhead; it routes connections using transport-level information without understanding application content.

The algorithm decision follows the traffic shape. Use round-robin when requests have roughly similar costs and servers have comparable capacity. If request duration varies or connections are long-lived, least-connections is a better starting point, although connection count remains only a proxy for actual load. Weighted routing handles heterogeneous capacity: during a rolling upgrade, for example, a higher-capacity server can receive a larger share while older servers remain in the pool. Consistent hashing is the choice when locality matters—such as mapping a session key or cache key to the same backend—because membership changes remap only a fraction of keys rather than reshuffling every key.

The decision flips when application visibility or state is non-negotiable. L4 is the right tool for simple TCP/UDP traffic where content-based routing is unnecessary; it cannot distinguish requests inside a multiplexed application connection. L7 earns its cost when you need HTTP-aware routing, security policy, or protocol awareness—for example, separating API paths or routing HTTP/2 and gRPC requests by application attributes. In a stateful WebSocket service, round-robin can send successive connections to different servers without preserving the state each connection needs; affinity or consistent hashing keeps the connection's identity near its state. In an interview, name the layer at the edge and between tiers, then tie the algorithm to request variance, capacity, connection lifetime, and locality.

Failure Mode: Shallow Checks and Flapping Backends

A shallow GET /healthz can return 200 while the instance is unable to serve real traffic. The process may be alive even though its database connection pool, cache, downstream dependency, or thread pool is exhausted. Passive checking discovers that only after a real request fails or times out, so the first users pay the failure cost before the balancer removes the instance.

Active checks move detection earlier, but their timeout is a stability trade-off. A probe timeout that is shorter than a normal load spike produces false failures; a timeout that is too long leaves an unhealthy instance serving traffic while probes hang. Azure's health-check guidance gives the relationship as App < Probe < Period x Threshold: the application should time out before the probe, and the probe should finish before the failure window closes.

Flapping makes the problem worse. Without separate failure and success thresholds, a backend can be removed and re-admitted too readily as probe results alternate. The backend repeatedly enters and leaves rotation, while traffic and connection load shift among the remaining servers. Use hysteresis: require a failure threshold before removal and M consecutive successes before re-admission. In one example readiness configuration, three failures at five-second intervals produce a 15-second removal window, while two successes at five-second intervals produce a 10-second recovery window; use such thresholds as design parameters rather than universal defaults. That trades slower recovery for a less trigger-happy pool.

The checkout case shows why dependency-aware readiness matters. If /healthz performs a lightweight SELECT 1 against the database replica and the replica becomes read-only during failover, the endpoint returns 503. The balancer can then stop sending write traffic to that node before customers encounter checkout errors. Keep dependency probes bounded and lightweight: checking every downstream system can itself consume the resources the instance needs to recover.

Failure Mode: The Load Balancer Becomes the Outage

A load balancer can become the outage it was supposed to prevent. If the only frontend process fails, clients can no longer reach any healthy backend—even when every backend is still serving traffic. The failure sequence is straightforward: the frontend address points at the failed balancer, connection attempts stop there, and the backend fleet receives no new work. Operators see frontend connection failures while backend CPU and request volume fall, which can misleadingly make the backend tier look healthy.

Remove that single failure domain by running a redundant pair. In an active-passive arrangement, one balancer owns the shared virtual IP (VIP) and forwards traffic; its peer monitors it and takes ownership after failure. In active-active, both balancers serve traffic, so losing one reduces frontend capacity rather than removing the path entirely. VRRP keeps the VIP associated with the current owner during failover, allowing clients to continue using one frontend address instead of learning a new address.

A practical topology puts an external L7 load balancer—or an API gateway that performs that role—at the internet edge. It can route HTTP traffic to the appropriate service tier. Internal L4 balancers then distribute transport traffic between service tiers when lower-latency routing is more important than application-aware routing. Azure's guidance notes that an effective setup often uses more than one type of load-balancing solution at different places in the workload; Google Cloud documents external load balancing for internet clients and internal load balancing for clients inside the cloud.

Redundancy must also include backend membership. Each balancer should apply the health-check policy to its own pool and remove a backend when the check says it cannot serve traffic. Otherwise, a partial backend failure can continue returning errors through both frontend nodes and become a full service outage. The pair protects the entry point; health-based removal protects the destinations.

How to Defend the Design in an Interview

A strong design answer names the load balancer’s position before naming the product. For internet traffic, place an external L7 balancer or API gateway at the edge when you need HTTP routing, TLS termination, or session-aware behavior. Between internal service tiers, an L4 balancer may be the better fit when routing by IP and TCP or UDP port is sufficient and you want lower overhead. State the boundary explicitly: “The edge balancer routes HTTP requests by path; internal traffic uses L4 because the services do not need payload-aware routing.”

Then connect the algorithm to the traffic pattern. Use round-robin for similarly sized, short-lived requests. Use least-connections or least-outstanding-requests when request cost varies or connections are long-lived. Use weights when backend capacity differs, such as during a rolling upgrade. For a distributed cache or another stateful partitioned workload, use consistent hashing so the same key reaches the same node and membership changes remap only a fraction of keys.

Health checks should answer “can this backend serve this class of traffic?”, not merely “is the process running?” Describe the probe, its interval and failure threshold, then name the dependencies it verifies: for example, database connectivity, cache reachability, or a required downstream service. Also state the recovery rule, such as requiring consecutive successful probes before re-admission, so a flapping backend does not thrash in and out of rotation.

Finally, explain the session model. Keep application backends stateless where possible and put shared session state in a shared cache such as Redis; then any healthy backend can serve a request. A stateful WebSocket service is different: round-robin does not provide the affinity needed to keep successive WebSocket connections or related state on the required backend. Use consistent hashing or cookie-based affinity to keep a client on the backend holding its connection state. For availability, run the balancer as an active-active or active-passive pair and preserve the shared virtual IP with VRRP.