Load Balancing Fundamentals

Before we dive into the specifics of tools like HAProxy, Nginx, or cloud-native solutions, we need to build a general overview of what load balancing actually is. At its simplest, load balancing answers a single question: When a client wants to talk to "the service," which specific server should actually handle that request?

Why Do We Need It?

You need load balancing the moment a single server can no longer (or should no longer) handle your traffic alone. It provides three core pillars for your infrastructure:

• Scalability:The ability to spread traffic across many servers instead of one.
• Availability:If one server fails, the load balancer shifts traffic to the healthy servers.
• Flexibility: You can add, remove, or replace backend servers without the client ever noticing.

Everything else - algorithms, health checks, SSL termination, routing rules - exists in service of that one core job.

The Four Layers of Traffic Management

It helps to think of traffic control as happening across a few layers, each solving a different part of the problem:

1. DNS-Based Balancing

The client's very first step is resolving a domain name to an IP address. DNS-based approaches (round-robin DNS, GeoDNS, services like Route 53) can return different IPs to different clients or regions.

• Strength: simple, works before a connection is even made, great for geographic routing.
• Weakness: DNS has no real awareness of server health, and changes propagate slowly due to caching (TTLs). It's coarse - a blunt instrument, not a scalpel.

2. Dedicated Load Balancers

This is the layer most people mean when they say "load balancer" - a piece of software or hardware sitting in front of a pool of servers, making a routing decision for every connection or request. This is where tools like HAProxy, Nginx, and cloud load balancers (AWS ALB/NLB, GCP LB) live.

Within this layer, there's an important sub-distinction:

• Layer 4 (Transport Layer):Often referred to in configuration as mode tcp, this handles raw connections based on IP addresses and ports. It is fast and protocol-agnostic because it doesn't "look" inside the packets - it just forwards them.
• Layer 7 (Application Layer): Referred to as mode http, this level understands application protocols like HTTP or WebSockets. It can make smart routing decisions based on URL paths (e.g., sending /api to one group and /static to another), headers, or cookies.

3. Reverse Proxies

A reverse proxy sits in front of backend servers and forwards client requests to them - which sounds identical to a load balancer, and the two concepts genuinely overlap. The distinction is more about primary purpose than mechanism:
A reverse proxy's core job is often things like SSL termination, caching, compression, and hiding backend topology - load balancing is frequently just one of its features.
A load balancer's core job is distributing traffic - some reverse proxies (like Nginx or Caddy) do this well enough to be used as a full load balancing solution.

In practice, most modern tools blur this line entirely. Nginx, Caddy, and even HAProxy can all reasonably be called reverse proxies and load balancers depending on how you configure and use them.

4. Service Mesh / Sidecar Proxies

In containerized, microservices-heavy environments (think Kubernetes), traffic management often moves inside the cluster itself. Tools like Envoy, Istio, and Linkerd deploy a small proxy alongside every service instance, handling service-to-service traffic with fine-grained control, dynamic service discovery, and built-in observability (tracing, metrics).
This is a fundamentally different operating model from a single load balancer sitting at the "front door" - it's load balancing distributed throughout the system rather than concentrated at one edge.

Choosing the Right Algorithm

How does the load balancer choose which server gets the next request? This is determined by the load balancing algorithm.

• Round Robin: The default choice; it simply takes turns, sending requests to each server in the list one by one.
• Least Connections (leastconn): Sends the next request to the server with the fewest active connections, making it ideal for long-lived sessions or varying request weights.
• Source: Uses a hash of the client's IP address to ensure a specific user always connects to the same backend server.

Essential Features: Health Checks and Persistence

A "sane" load balancer doesn't just blindly send traffic; it monitors the environment.

• Health Checks:These are proactive probes (like a ping or an HTTP request) sent periodically to ensure a server is still reachable and operational. If a server fails, it is automatically removed from the rotation until it recovers.
• Sticky Sessions (Persistence):Some applications (like those with shopping carts or logins stored in local memory) require a user to stay on the same server for the entire duration of their visit. This is often achieved through cookies or session identifiers.
  

Avoiding the Single Point of Failure

If your load balancer sits in front of all your servers, what happens if the load balancer itself dies?. Without redundancy, the load balancer becomes a Single Point of Failure (SPOF).

To solve this, production environments use High Availability (HA) setups:

• Active/Passive: Two load balancers exist, but only one handles traffic. A "Floating IP" is managed by a protocol like VRRP (often via Keepalived); if the active node fails, the IP automatically shifts to the passive node
• Active/Active: Both load balancers are engaged simultaneously, often used for higher throughput or regional clusters.