DevOps School

---

Quick Definition

A load balancer is a network component or service that distributes incoming traffic across multiple backend targets to improve availability, performance, and resilience.

Analogy: Like an airport ground controller routing arriving flights to open gates so no single gate becomes overwhelmed.

Formal technical line: A load balancer performs request distribution and health-aware routing according to configured algorithms and policies, often operating at Layer 4 (transport) or Layer 7 (application).

Multiple meanings:

• Most common: Network or application service that distributes client requests to multiple servers or services.
• Other meanings:
• Hardware appliance providing traffic distribution and offload.
• Cloud-managed service that abstracts routing and scaling for tenants.
• Software proxy/load-distribution library embedded in applications.

---

What is load balancer?

What it is / what it is NOT

• What it is: A traffic director that routes client requests to a pool of healthy backends using rules, algorithms, and health checks.
• What it is NOT: A full application firewall, identity provider, or general-purpose reverse proxy (though it can include aspects of these).

Key properties and constraints

• Algorithm types: round-robin, least-connections, weighted, header-based, latency-aware.
• Layers: L4 (IP/TCP/UDP), L7 (HTTP/HTTPS/gRPC/WebSocket).
• Health checks: TCP, HTTP(S), gRPC probes with configurable thresholds.
• Session affinity: optional sticky sessions using cookies, source IP, or tokens.
• SSL/TLS termination: can terminate or pass-through.
• Performance constraints: CPU, memory, and network I/O limits; connection tracking table sizes.
• Consistency constraints: sticky sessions or hashing methods can affect cache locality and scaling.

Where it fits in modern cloud/SRE workflows

• Edge layer handling ingress traffic and enforcing TLS and routing policies.
• Service mesh or L7 proxies providing east-west balancing inside clusters.
• Autoscaling trigger point and can integrate with orchestration APIs.
• Observability pivot: central place for latency, error, and traffic metrics used by SREs.
• Incident playbooks often start with load balancer health, configuration drift, or DNS issues.

Diagram description (visualize in text)

• Clients -> Public edge load balancer (TLS) -> WAF / CDN optional -> Internal load balancer -> Service pool (VMs, containers, serverless endpoints) -> Databases and caches. Health checks flow back from load balancer to services; metrics flow from load balancer to monitoring; autoscaler reads metrics and adjusts service pool.

load balancer in one sentence

A load balancer is a traffic control point that distributes client requests across multiple backends while enforcing health checks, routing rules, and performance policies.

load balancer vs related terms (TABLE REQUIRED)

ID	Term	How it differs from load balancer	Common confusion
T1	Reverse proxy	Focuses on request routing and caching not always on load distribution	Confused as same when functions overlap
T2	API gateway	Adds auth, rate limiting, transformation on top of routing	People expect LB to do API management
T3	Service mesh	Provides per-service proxies and telemetry in-cluster	Thought to replace external LB
T4	CDN	Caches and serves static content from edge nodes	Often mistaken for LB for global routing
T5	NAT gateway	Translates addresses not balancing based on health	Users mix up IP translation with distribution
T6	DNS load balancing	Uses DNS responses for distribution lacking health granularity	Assumed to be real-time LB

Row Details (only if any cell says “See details below”)

• None

---

Why does load balancer matter?

Business impact

• Revenue: Ensures customer-facing services stay responsive; outages or high latency can reduce conversions and revenue.
• Trust: Consistent availability improves customer trust and retention.
• Risk: Misconfigured or under-provisioned LBs increase single points of failure and regulatory exposure for availability SLAs.

Engineering impact

• Incident reduction: Health checks and automatic rerouting typically lower MTTR by avoiding routing to unhealthy hosts.
• Velocity: Centralized routing and configuration APIs enable safer deployment patterns (canaries, blue-green).
• Complexity trade-off: Adds operational overhead; requires observability and testing.

SRE framing

• SLIs/SLOs: Availability, request latency, and error rate measured at the load balancer boundary.
• Error budgets: Drive decisions such as releasing new routing rules or scaling pools.
• Toil: Repetitive manual changes should be automated (infrastructure as code).
• On-call: Load balancer incidents are high-severity and usually page network and platform owners.

What commonly breaks in production

1. Misrouted traffic due to incorrect routing rules or host header mismatches.
2. TLS certificate expiration or mismatched ciphers causing handshake failures.
3. Health checks misconfigured leading to healthy hosts marked unhealthy and traffic storms.
4. Session stickiness causing uneven load and resource hot spots.
5. Connection table exhaustion under DDoS or traffic spike events.

---

Where is load balancer used? (TABLE REQUIRED)

ID	Layer/Area	How load balancer appears	Typical telemetry	Common tools
L1	Edge	Public LB terminating TLS and routing hosts	request rate latency TLS handshakes	Cloud LBs, HAProxy
L2	Network	L4 TCP/UDP distribution and NAT	connection count errors	F5, metalLB
L3	Service	Ingress controllers and sidecars	per-route latencies success rate	Envoy, Traefik
L4	App	Application-level routing and transformations	HTTP status distribution	API gateways
L5	Data	Load distribution for DB proxies and caching	connection wait time error rates	PgBouncer, ProxySQL
L6	Kubernetes	Ingress, Service type LoadBalancer, IngressController	pod backend latency health	kube-proxy, cloud providers
L7	Serverless/PaaS	Managed LBs mapping custom domains to functions	cold starts invocations errors	Provider-managed LBs
L8	CI/CD	Test routing for canaries and blue-green	deployment success A/B metrics	Feature flags, LB API
L9	Observability	Collection point for request traces and logs	request traces sampled logs	Logging, APM
L10	Security	Enforce rate limits WAF and IP filters	blocked rate challenge rates	WAFs, LB rules

Row Details (only if needed)

• None

---

When should you use load balancer?

When it’s necessary

• Multiple identical backend instances exist and you need availability and capacity distribution.
• Public or internal endpoints require TLS termination and path/host-based routing.
• Autoscaling or rolling deployments are used; an LB provides seamless backend churn.

When it’s optional

• Single-instance services with low traffic and no availability requirement.
• Very simple internal tools where DNS round-robin suffices for tolerance.

When NOT to use / overuse it

• Avoid using LB for fine-grained access control that belongs in application logic.
• Don’t use LB session stickiness as the primary method to preserve state; use distributed caches or session stores instead.
• Avoid adding an LB layer for micro-optimizations that add latency and operational load.

Decision checklist

• If you need TLS termination and multi-backend failover -> use an LB.
• If you need per-request auth, transformation, or API composition -> consider API gateway plus LB.
• If you have simple low-throughput service inside a trusted network -> DNS + client retry might suffice.

Maturity ladder

• Beginner: Single cloud-managed LB terminating TLS and routing by host.
• Intermediate: Ingress controllers inside Kubernetes with health checks and canary routing.
• Advanced: Global traffic management with regional LBs, active-active failover, and programmable routing via service mesh.

Example decision for a small team

• Small SaaS with a single service: use managed cloud LB with autoscaling group + basic health checks.

Example decision for large enterprise

• Global web presence: use regional cloud LBs + global traffic manager + CDN + active-active backends with cross-region health and failover.

---

How does load balancer work?

Components and workflow

• Listener: Accepts client connections on ports/protocols.
• Routing rules: Match host/path/headers to backend pools.
• Backend pools/target groups: Set of servers with weights and health check settings.
• Health checks: Periodic probes determining backend availability.
• Session management: Sticky sessions or stateless forwarding.
• Metrics & logs: Request counts, latencies, error rates, TLS stats.
• Control plane: API/UI to modify rules and backends.
• Data plane: High-performance forwarding process handling packets/connections.

Data flow and lifecycle

1. Client connects to LB listener (e.g., port 443).
2. LB selects backend target using configured algorithm.
3. LB opens backend connection or forwards request.
4. Backend responds; LB forwards response to client.
5. Health checks run in parallel and update backend state.
6. Metrics emitted to monitoring and can trigger autoscaling.

Edge cases and failure modes

• Backend slow response causing head-of-line blocking in L4 connection pooling.
• Inconsistent session hashing after scaling events leads to cache misses.
• Health check flapping marking healthy hosts unhealthy, causing oscillation.
• DNS TTL mismatches with LB changes causing stale client routing.

Short practical examples (pseudocode)

• Pseudocode for weighted round-robin:
• Maintain weight counters per target, select highest effective weight, decrement, rotate.
• Health-check policy:
• Send GET /health every 5s; mark unhealthy after 3 failures; recover after 2 successes.

Typical architecture patterns for load balancer

• Edge terminated LB + CDN: Use when global caching and TLS offload are needed.
• L4 pass-through LB + internal L7 proxy: Use when end-to-end TLS is required and application routing is done inside.
• Sidecar/Service-mesh based L7 balancing: Use when per-service telemetry and fine-grained policies are needed.
• Global DNS-based LB + regional active-active LBs: Use for multi-region failover and low-latency routing.
• Host/path-based ingress controller in Kubernetes: Use when multiple services share a cluster IP and domain.

Failure modes & mitigation (TABLE REQUIRED)

ID	Failure mode	Symptom	Likely cause	Mitigation	Observability signal
F1	Health check flapping	Backends repeatedly marked unhealthy	Flaky checks or resource spikes	Harden checks add thresholds	spike in check failures
F2	TLS handshake errors	Clients fail to connect with TLS errors	Expired cert or cipher mismatch	Rotate certs update ciphers	TLS handshake failure rate
F3	Connection table exhaustion	New connections dropped or slowed	High concurrent connections or DDoS	Increase tables or rate-limit	SYN queue growth
F4	Bad routing rules	404 or wrong backend responses	Misconfigured host/path mapping	Revert rule update validate before deploy	Surge in 404 mismatches
F5	Session imbalance	Some instances overloaded	Improper affinity or hashing	Reconfigure affinity or use stateless sessions	uneven backend CPU usage
F6	Control plane lag	Config changes delayed applying	API rate limits or failing agents	Retry with backoff monitor agent	config apply latency
F7	Certificate key compromise	Risk of MITM or unauthorized access	Private key leaked	Rotate keys revoke old certs	Unexpected cert issuer alerts

Row Details (only if needed)

• None

---

Key Concepts, Keywords & Terminology for load balancer

Glossary (40+ terms)

• Algorithm — The rule used to select a backend — Determines distribution fairness — Pitfall: choosing wrong algorithm for sticky needs.
• Anycast — Single IP announced from multiple locations — Enables geo routing — Pitfall: stateful sessions may break.
• Backend pool — Group of targets serving requests — Abstracts instances for routing — Pitfall: mixing incompatible versions.
• Backend weight — Relative share of traffic for targets — Controls capacity distribution — Pitfall: wrong weights cause overload.
• Blue-green deploy — Two parallel environments for zero-downtime deploys — Simplifies rollback — Pitfall: stale data migrations.
• Canary release — Gradual traffic shift to new version — Limits blast radius — Pitfall: insufficient traffic to detect bugs.
• Client IP preservation — Passing original client IP to backend — Important for logging and ACLs — Pitfall: NAT hides client address.
• Connection draining — Let existing sessions finish before removing backend — Prevents abrupt failures — Pitfall: misconfigured timeout allows new sessions.
• Consistent hashing — Map keys to backends with minimal reshuffle — Useful for caching affinity — Pitfall: changing ring nodes invalidates caches.
• Control plane — Management API/UI for LB config — Centralizes changes — Pitfall: single point of config failure.
• Default backend — Fallback target for unmatched requests — Provides predictable behavior — Pitfall: accidentally routing all to default.
• DNS TTL — How long DNS clients cache LB IP — Affects failover speed — Pitfall: long TTLs delay rollbacks.
• DDoS protection — Mechanisms to absorb or block malicious traffic — Protects LBs from overload — Pitfall: false positives blocking legit users.
• Edge routing — First hop for external traffic — Enforces TLS and access controls — Pitfall: misconfig leading to open endpoints.
• Endpoint — Individual server, pod, or function handling requests — Unit of scaling — Pitfall: inconsistent endpoint config.
• Fail-open vs fail-closed — Behavior when a dependency fails — Choice impacts availability vs security — Pitfall: choosing wrong default.
• Flow control — Mechanism to prevent overload under pressure — Protects backends — Pitfall: dropping connections without retry.
• Health probe — Periodic check to validate a backend — Drives routing decisions — Pitfall: endpoint heavy checks increase load.
• HAProxy — Popular open-source LB — Feature-rich L4/L7 with ACLs — Pitfall: complex config if misused.
• Heartbeat — Low-level liveness signal — Used in HA designs — Pitfall: misinterpretation of delayed heartbeats.
• Horizontal scaling — Add more instances to pool — Common scaling method — Pitfall: stateful components don’t scale linearly.
• HTTP/2 multiplexing — Multiple requests per connection — Reduces connections cost — Pitfall: backend HTTP/2 support mismatch.
• Ingress controller — Kubernetes component implementing L7 routing — Integrates cluster routing — Pitfall: mismatched annotations or CRDs.
• IPVS — Kernel-level L4 proxying used by kube-proxy — High performance L4 balancing — Pitfall: operational complexity on upgrades.
• Latency-aware routing — Send requests to lowest-latency backends — Improves performance — Pitfall: noisy latency signals misroute traffic.
• Layer 4 (L4) — Transport-level balancing (TCP/UDP) — Fast and protocol-agnostic — Pitfall: less visibility into HTTP semantics.
• Layer 7 (L7) — Application-level balancing (HTTP) — Enables host/path routing and header rules — Pitfall: higher CPU cost.
• Least connections — Algorithm favoring less-busy servers — Useful for long-lived connections — Pitfall: poor for highly variable request cost.
• Load shedding — Intentionally drop or reject requests to protect system — Preserves core functionality — Pitfall: needs graceful handling upstream.
• Mutual TLS (mTLS) — Two-way certificate auth — Provides strong identity — Pitfall: certificate management complexity.
• NAT gateway — Translates source addresses outbound — Differs from LB role — Pitfall: confusing address translation with distribution.
• NGINX — Popular web server used as L7 LB — Flexible and performant — Pitfall: complex cache and rewrite rules cause bugs.
• Observability — Metrics, logs, traces around LB behavior — Essential for diagnosis — Pitfall: sampling hiding rare failure modes.
• Packet per second (PPS) — Measure of LB throughput at packet level — Important for UDP and small payloads — Pitfall: ignoring PPS can overload CPU.
• Proxy protocol — Preserves source IP across proxy layers — Helps backend identify client — Pitfall: must be enabled both sides.
• Rate limiting — Controls requests per client or token — Mitigates abuse — Pitfall: poor thresholds block legitimate traffic.
• Session affinity — Sticky sessions to same backend — Useful for legacy apps — Pitfall: uneven load and single-host failure.
• Service mesh — Distributed proxy architecture for service-to-service LB — Adds telemetry and policy — Pitfall: complexity and increased latency.
• SSL offload — Terminate TLS at the LB to reduce backend load — Simplifies cert management — Pitfall: backend must accept plain traffic or re-encrypt.
• TCP keepalive — Low-level connection liveness setting — Helps detect dead clients — Pitfall: misconfigured values lead to resource leaks.
• Weighted least connection — Combination algorithm using weights and active connections — Balances capacity and load — Pitfall: complexity in tuning.

---

How to Measure load balancer (Metrics, SLIs, SLOs) (TABLE REQUIRED)

ID	Metric/SLI	What it tells you	How to measure	Starting target	Gotchas
M1	Request rate	Traffic volume per second	Count requests at LB boundary	Baseline by traffic pattern	Bursts distort averages
M2	95p latency	User-facing latency distribution	Measure request duration at LB	95p under SLO-defined ms	Include TLS handshake time
M3	Error rate	Fraction of 4xx/5xx at LB	Error count / total requests	<1% depending on SLA	Upstream vs LB errors mixed
M4	TLS handshake failures	TLS negotiation problems	Count handshake errors	Near zero	Client ciphers cause failures
M5	Backend healthy ratio	Percent of healthy targets	Healthy target count / total	>90% healthy typical	Misconfigured checks reduce ratio
M6	Connection count	Active connections on LB	Track concurrent connections	Depends on app load	Long connections skew capacity
M7	Time to failover	How fast traffic moves from bad backends	Measure time from failure to restored traffic	<30s typical for internal LBs	DNS TTL affects global failover
M8	5xx spike rate	Backend error surge visibility	5xx count/time window	Alert on sustained rise	Short spikes may be noise
M9	SYN flood rate	Signs of connection storms	Monitor SYNs/sec and drops	Alert threshold by baseline	Requires kernel metrics
M10	Health check latency	Probe response time	Average probe duration	Low ms for fast checks	Heavy checks add backend load
M11	Backend response time	Backend processing latency	Measure backend duration at LB	Align with app SLOs	LB adds minimal overhead
M12	Drop/reject rate	Requests rejected by LB policies	Rejected count / total	Minimize rejections	Misconfigured rules cause false rejects

Row Details (only if needed)

• None

Best tools to measure load balancer

Tool — Prometheus + Exporters

• What it measures for load balancer: Metrics for request rates, latencies, connection counts.
• Best-fit environment: Kubernetes, self-managed LBs, cloud-native stacks.
• Setup outline:
• Install exporters or LB native metric endpoints.
• Configure scrape jobs and relabeling.
• Define recording rules for SLI windows.
• Create alerts for threshold breaches.
• Strengths:
• Powerful query language and ecosystem.
• Works well with Kubernetes.
• Limitations:
• Long-term storage and scaling require remote write or adapters.
• Requires ops effort to maintain.

Tool — Managed cloud monitoring (vary by provider)

• What it measures for load balancer: Provider-specific LB metrics and logs.
• Best-fit environment: Cloud-managed LBs in public clouds.
• Setup outline:
• Enable LB metrics and logging in cloud console.
• Configure export to central monitoring.
• Set alerts on provided metrics.
• Strengths:
• Integrated with provider features.
• Minimal setup for basic telemetry.
• Limitations:
• Metrics granularity and retention vary / Not publicly stated.

Tool — Datadog

• What it measures for load balancer: Aggregated LB metrics, traces, and dashboards.
• Best-fit environment: Hybrid cloud and multi-service environments.
• Setup outline:
• Install agents or integrate cloud provider.
• Import LB dashboards and configure monitors.
• Enable tracing for request-level details.
• Strengths:
• Rich dashboards and out-of-the-box monitors.
• Correlates metrics and traces.
• Limitations:
• Cost at scale and depends on sampling choices.

Tool — Elastic Observability

• What it measures for load balancer: Logs, metrics, traces from LBs and backends.
• Best-fit environment: Organizations using Elastic stack for observability.
• Setup outline:
• Ship LB logs/metrics via beats or ingest pipelines.
• Create dashboards and alerting rules.
• Use traces to link LB to services.
• Strengths:
• Flexible log processing and search.
• Limitations:
• Requires sizing for index storage.

Tool — OpenTelemetry + backend

• What it measures for load balancer: Traces and metrics enabling end-to-end request visibility.
• Best-fit environment: Distributed systems with instrumented services.
• Setup outline:
• Add instrumentation on LB or ingress proxy.
• Export to chosen backend.
• Define SLI calculations using traces.
• Strengths:
• Standardized telemetry across stack.
• Limitations:
• Requires implementation on proxy or sidecars; not always present.

Recommended dashboards & alerts for load balancer

Executive dashboard

• Panels:
• Global availability percentage for all public endpoints.
• 95th and 99th percentile latency trends.
• Top error codes and traffic by region.
• Capacity utilization trend.
• Why: High-level view for execs and platform owners to spot service health.

On-call dashboard

• Panels:
• Real-time error rate and request rate.
• Backend healthy ratio and target list with statuses.
• Active alerts and incident timeline.
• Top slow endpoints and recent 5xx traces.
• Why: Prioritized data for responders to triage fast.

Debug dashboard

• Panels:
• Live request traces for recent errors.
• Connection table utilization and SYN stats.
• Detailed per-backend CPU, memory, latency.
• Health check success/failure timeline.
• Why: Supports deep investigation and root cause analysis.

Alerting guidance

• Page vs ticket:
• Page for high-severity SLO breaches (availability, large error spike).
• Create ticket for lower-priority degradations or capacity warnings.
• Burn-rate guidance:
• Use burn-rate to escalate when error budget is being consumed >3x expected.
• Noise reduction tactics:
• Deduplicate alerts by grouping by LB and region.
• Use suppression windows for routine maintenance.
• Use composite alerts combining multiple signals to reduce false positives.

---

Implementation Guide (Step-by-step)

1) Prerequisites – Inventory endpoints, TLS requirements, and expected traffic patterns. – Define SLOs and error budgets for services behind LB. – Provision VPC/subnet and security groups/ACLs.

2) Instrumentation plan – Enable LB metrics and logs. – Ensure request tracing spans LB to backends. – Export health check and config-change events.

3) Data collection – Centralize logs, metrics, and traces. – Ensure retention aligns with postmortem requirements.

4) SLO design – Define availability and latency SLOs at LB boundary. – Map SLO targets to business objectives and error budgets.

5) Dashboards – Build executive, on-call, and debug dashboards as earlier described.

6) Alerts & routing – Configure alert thresholds, routing for on-call, and escalation policies. – Use runbook links in alert messages.

7) Runbooks & automation – Create playbooks for common failures (TLS, health checks, config rollback). – Automate LB config via IaC and CI pipelines with validation steps.

8) Validation (load/chaos/game days) – Run load tests and simulate backend failures. – Execute chaos experiments: kill targets, tweak health checks, and verify failover.

9) Continuous improvement – Review incidents, refine health checks, adjust thresholds and automation.

Checklists

Pre-production checklist

• TLS certs uploaded and validated.
• Health checks defined and pass on test backends.
• Route rules tested in staging with full traffic patterns.
• Metrics and logging confirmed in monitoring.
• IaC templates reviewed and tagged.

Production readiness checklist

• Autoscaling policy validated under load.
• Rate limits and WAF rules reviewed.
• Runbooks published and on-call trained.
• Alerting and suppression rules tested.

Incident checklist specific to load balancer

• Verify LB config changes in audit log.
• Check health check failure logs and timestamps.
• Validate backend process and resource usage.
• Rollback recent LB rule changes if indicated.
• Re-route traffic via alternate LB or region if necessary.

Examples

• Kubernetes: Implement Ingress controller with readiness and liveness probes; deploy Service type LoadBalancer mapped to cloud LB; test canary via Ingress rules and augment with Istio or Envoy for advanced routing.
• Managed cloud service: Use cloud LB with target groups, attach autoscaling group, configure health checks to an application /live endpoint, and automate via cloud IaC (templates/terraform); verify endpoints in staging before promoting.

What “good” looks like

• Health checks stable with >95% healthy targets.
• Error budget consumption within plan.
• Automated rollbacks for LB misconfiguration validated.

---

Use Cases of load balancer

1) Public web storefront – Context: High traffic consumer site. – Problem: Need high availability and TLS offload. – Why LB helps: Distributes traffic and terminates TLS with health-aware failover. – What to measure: Availability, 95p latency, TLS errors. – Typical tools: Cloud LBs plus CDN.

2) Kubernetes multi-tenant cluster ingress – Context: Different teams host services in same cluster. – Problem: Router isolation, path-based routing, and quota enforcement. – Why LB helps: Single entrypoint with rules and authentication. – What to measure: Per-tenant request rate and error rates. – Typical tools: Ingress controller + RBAC.

3) Microservice east-west balancing – Context: Numerous internal services with dynamic scaling. – Problem: Need fine-grained routing and tracing. – Why LB helps: Service mesh proxies provide balanced and observable traffic. – What to measure: Service-to-service latency and retries. – Typical tools: Envoy, service mesh.

4) Database proxying – Context: Pooling connections to a database. – Problem: Backend DB limited concurrent connections. – Why LB helps: Distribute and pool connections effectively. – What to measure: Connection wait times and saturation. – Typical tools: PgBouncer, ProxySQL.

5) Global failover – Context: Multi-region deployments for resilience. – Problem: Route users to nearest healthy region. – Why LB helps: Regional LBs combined with global traffic manager handle failovers. – What to measure: Time to failover and cross-region latency. – Typical tools: Global traffic manager + regional LBs.

6) Canary deployments – Context: Rolling out new service version. – Problem: Need safe incremental exposure. – Why LB helps: Direct percentage of traffic to canary and monitor. – What to measure: Error spike correlation and business metrics. – Typical tools: API gateway, LB weighted routing.

7) Serverless function routing – Context: Functions behind custom domains. – Problem: Mapping custom domains and TLS to functions. – Why LB helps: Fronts serverless endpoints and handles routing. – What to measure: Invocation latency and cold start rate. – Typical tools: Cloud-managed LBs and function gateway.

8) API aggregation – Context: Composite API that calls multiple backends. – Problem: Need request routing and timeouts. – Why LB helps: Centralize routing policies and enforce timeouts and retries. – What to measure: Aggregation latency and partial failure rate. – Typical tools: API Gateway + LB.

9) DDoS mitigation – Context: Public-facing high-value services. – Problem: Malicious traffic causing outages. – Why LB helps: Throttle, rate-limit, and route through DDoS protection. – What to measure: SYN flood rate and dropped requests. – Typical tools: WAF, LB with rate limiting.

10) Edge compute routing for low-latency apps – Context: Interactive apps with global users. – Problem: Need region-aware routing and minimal latency. – Why LB helps: Edge LBs route to nearest compute nodes. – What to measure: User-perceived latency and P99. – Typical tools: Anycast LBs and edge proxies.

---

Scenario Examples (Realistic, End-to-End)

Scenario #1 — Kubernetes: Canary rollout with ingress

Context: A team runs a microservice on Kubernetes and needs to roll out v2 gradually. Goal: Safely route 5% traffic to v2 while monitoring. Why load balancer matters here: LB must support weighted routing to target v2 pods and rapidly shift traffic if errors escalate. Architecture / workflow: Client -> Cloud LB -> Ingress Controller -> Service selector weights -> Pod sets. Step-by-step implementation:

• Create new Deployment v2 and Service versioned label.
• Configure Ingress with annotation for weighted routing or use service mesh virtual service for traffic split.
• Add health checks for v2 and monitoring alerts for errors.
• Start with 5% weight, monitor for 24h, increase if stable. What to measure: Error rate on v2 vs baseline, latency tail, business metrics. Tools to use and why: Ingress + Istio or Envoy for precise splits and telemetry. Common pitfalls: Missing readiness probes causing LB to route to unready pods. Validation: Run synthetic traffic and failure injection on v2 to verify rollback triggers. Outcome: Controlled rollout with ability to revert quickly upon anomalies.

Scenario #2 — Serverless/PaaS: Custom domain mapping to functions

Context: A marketing team needs custom domain for a set of serverless functions. Goal: Route HTTPS traffic to functions with custom TLS and origin health. Why load balancer matters here: LB abstracts domain/TLS and efficiently routes to function endpoints while collecting metrics. Architecture / workflow: Client -> Managed LB -> TLS termination -> Auth/Zones -> Function invoker. Step-by-step implementation:

• Provision cloud-managed LB and upload cert.
• Map domain to LB and configure path-based routing to function endpoints.
• Enable function cold-start monitoring and include retries. What to measure: Invocation latency, cold start rate, errors. Tools to use and why: Provider-managed LB simplifies TLS and scale. Common pitfalls: High cold start counts when LB health probes are aggressive. Validation: Spike load tests to ensure scaling and routing behavior. Outcome: Stable custom-domain routing for serverless functions.

Scenario #3 — Incident-response/postmortem: Sudden 5xx spike

Context: Production site experiences 5xx spike and partial outage. Goal: Identify root cause and restore service. Why load balancer matters here: LB metrics indicate whether errors originate at LB, upstream, or due to routing changes. Architecture / workflow: LB logs -> monitoring -> on-call investigates backend and LB config. Step-by-step implementation:

• Check recent LB config changes and audit logs.
• Verify backend health checks and resource usage.
• If misconfiguration, revert using IaC.
• If backend failure, drain affected targets and shift traffic. What to measure: 5xx by backend, health check failures, time to failover. Tools to use and why: Centralized logging and tracing to correlate requests and errors. Common pitfalls: Jumping to restart backends without checking LB rules. Validation: Postmortem shows root cause and action items for checks and automation. Outcome: Restored service and improved guardrails to prevent recurrence.

Scenario #4 — Cost/performance trade-off: SSL offload vs end-to-end TLS

Context: Team evaluates whether to offload TLS at LB or re-encrypt to backend. Goal: Balance CPU costs and security posture. Why load balancer matters here: Choice affects backend resource usage, latency, and certificate management. Architecture / workflow: Client -> LB (terminate TLS) -> re-encrypt or plain to backend. Step-by-step implementation:

• Measure CPU and latency impact of TLS on backends under load.
• Test re-encryption setup and certificate automation.
• Compare costs for compute vs managed LB TLS termination. What to measure: Backend CPU, added latency, operational overhead for certs. Tools to use and why: Load testing tools and monitoring to quantify trade-offs. Common pitfalls: Assuming minimal latency impact without measurement. Validation: A/B testing production-like load and cost modeling. Outcome: Informed decision aligning security and cost.

---

Common Mistakes, Anti-patterns, and Troubleshooting

List of mistakes with symptom -> root cause -> fix (selected highlights, 20 entries)

1. Symptom: Intermittent 502s -> Root cause: Backend listening port mismatch -> Fix: Verify service port and update LB target group.
2. Symptom: TLS handshake failures -> Root cause: Expired cert -> Fix: Rotate cert and automate renewal.
3. Symptom: Uneven load across instances -> Root cause: Sticky sessions enabled unnecessarily -> Fix: Disable affinity or move state to shared store.
4. Symptom: Slow failover -> Root cause: Long DNS TTL -> Fix: Reduce TTL or use global traffic manager with health checks.
5. Symptom: High 5xx after deploy -> Root cause: Canary incomplete health checks -> Fix: Add deeper readiness checks and circuit breaker.
6. Symptom: Control plane API errors -> Root cause: Rate-limited IaC executions -> Fix: Batch config updates and backoff retries.
7. Symptom: Connections dropped under peak -> Root cause: Connection table exhaustion -> Fix: Tune OS kernel and LB limits.
8. Symptom: Monitoring gaps -> Root cause: Metrics not exported from LB -> Fix: Enable LB metric endpoints and exporters.
9. Symptom: Unexpected geo routing -> Root cause: Anycast misconfiguration -> Fix: Validate BGP announcements and regional mapping.
10. Symptom: DDoS causing service degraded -> Root cause: No rate limiting or WAF rules -> Fix: Add rate limits and DDoS protection at edge.
11. Symptom: Health checks passing but users see errors -> Root cause: Health check probes not exercising real code paths -> Fix: Use realistic probes hitting downstream dependencies.
12. Symptom: Excessive retries -> Root cause: Tight retry policy at LB -> Fix: Lower retry attempts and add exponential backoff.
13. Symptom: Log noise and alert fatigue -> Root cause: Broad alert rules on transient errors -> Fix: Add aggregation windows and suppression during deploys.
14. Symptom: Insecure backend traffic -> Root cause: TLS termination without re-encryption where required -> Fix: Enable re-encrypt or mTLS for sensitive data.
15. Symptom: Canary never gets traffic -> Root cause: Weighted route misconfigured -> Fix: Validate routing weights and rollout config.
16. Symptom: Latency spikes for specific routes -> Root cause: Heavy transformations at LB (rewrites) -> Fix: Move heavy work to backend or precompute.
17. Symptom: Session loss after scaling -> Root cause: Consistent hashing reset after node change -> Fix: Use sticky cookies or external session store.
18. Symptom: Metrics not matching logs -> Root cause: Different sampling or aggregation windows -> Fix: Align collection windows and sampling settings.
19. Symptom: Over-reliance on LB for auth -> Root cause: Treating LB as API gateway -> Fix: Move auth to API gateway or service.
20. Symptom: Deployment rollback fails -> Root cause: Incomplete rollback plan for LB rules -> Fix: Implement IaC rollbacks and verify in staging.

Observability pitfalls (at least 5)

• Missing client IP in logs -> cause: no proxy protocol -> fix: enable proxy protocol and update backend parsing.
• Metrics rate mismatch -> cause: different scrape intervals -> fix: standardize scrape and retention.
• Trace sampling hides errors -> cause: low sampling rate -> fix: increase sampling for error traces.
• No link between LB metrics and backends -> cause: no trace propagation -> fix: add request IDs and trace headers.
• Alert context insufficient -> cause: alerts without runbook links or owner -> fix: include runbook URL and responder team in alert.

---

Best Practices & Operating Model

Ownership and on-call

• Assign platform team ownership for LB platform and integrate with SRE on-call rotations.
• Define escalation paths for DNS, network, and LB incidents.

Runbooks vs playbooks

• Runbooks: Step-by-step actions for common incidents (health-check failures, TLS rotation).
• Playbooks: High-level strategies for complex scenarios (regional failover, disaster recovery).

Safe deployments

• Use canary or blue-green patterns with LB weighted routing.
• Automate rollback and validate via synthetic tests.

Toil reduction and automation

• Automate LB configuration via IaC and CI pipelines.
• Automate certificate renewals and secret rotation.

Security basics

• Terminate TLS at the edge; re-encrypt if required for compliance.
• Apply WAF and rate limits at LB.
• Enforce least-privilege for LB control plane APIs.

Weekly/monthly routines

• Weekly: Review health-check flapping and top error paths.
• Monthly: Validate certificate expirations and rotate keys if needed.
• Quarterly: Run chaos tests and review capacity planning.

What to review in postmortems related to load balancer

• Timeline of LB config changes and related commits.
• Health-check configuration and sensitivity.
• Alert and SLO behavior during incident.
• Automation or rollout gaps to fix.

What to automate first

• Certificate rotation and monitoring.
• Health-check validation tests in CI.
• IaC validation and dry-run of LB changes.

---

Tooling & Integration Map for load balancer (TABLE REQUIRED)

ID	Category	What it does	Key integrations	Notes
I1	Cloud LB	Managed traffic distribution	Autoscaler DNS monitoring	Provider-specific features vary
I2	Ingress controller	L7 routing inside Kubernetes	Service mesh cert-manager	Use with kube-proxy and CRDs
I3	Service mesh	Per-service proxies and policies	Tracing APM CI/CD	Adds latency and complexity
I4	CDN	Edge caching and TLS	Origin LB WAF analytics	Use for static and edge cached content
I5	WAF	Protects from web attacks	LB rule integration SIEM	Requires tuning to reduce false positives
I6	Monitoring	Collects metrics and alerts	LB exporters traces logs	Core to SLO management
I7	Logging	Centralizes access and error logs	SIEM dashboards traces	Ensure structured logs for parsing
I8	Traffic manager	Global DNS and failover	Regional LBs health checks	Critical for multi-region routing
I9	DDoS protection	Mitigates large-scale attacks	Edge LB WAF rate limiting	May be managed service
I10	IaC	Declarative LB configuration	CI/CD pipeline monitoring	Enables reproducible changes

Row Details (only if needed)

• I1: Provider-managed LBs include autoscaling hooks and security features; specifics vary by vendor.
• I3: Service mesh replaces some LB features like routing but requires sidecar injection and policy management.

---

Frequently Asked Questions (FAQs)

H3: What is the difference between a load balancer and an API gateway?

A load balancer primarily distributes traffic and performs basic routing; an API gateway adds API management features like auth, rate limiting, and payload transformation.

H3: How do I choose L4 vs L7 balancing?

Choose L4 for high-throughput, low-latency TCP/UDP traffic and when you don’t need HTTP semantics; choose L7 if you need host/path routing, header-based logic, or payload inspection.

H3: How do I measure LB availability?

Measure availability as successful responses over total requests at the LB boundary, typically using 99.9% or similar SLO targets based on business needs.

H3: How do I perform canary releases with a load balancer?

Use weighted routing to split a small percentage of traffic to the canary backend and monitor errors and latency before increasing weight.

H3: How do I preserve client IP behind a proxy?

Enable proxy protocol or add X-Forwarded-For headers and ensure backends parse and log these headers.

H3: How do I secure my load balancer?

Terminate TLS at edge, implement WAF/rate limiting, use mTLS for internal traffic when needed, and restrict LB management access.

H3: What’s the difference between DNS load balancing and a proper LB?

DNS load balancing uses DNS responses to distribute traffic without real-time health checks; proper LBs perform health checks and immediate rerouting.

H3: What’s the difference between a reverse proxy and a load balancer?

A reverse proxy forwards client requests to servers and may include caching and transformations; a load balancer focuses on distributing load and health-aware routing.

H3: What’s the difference between a service mesh and a load balancer?

A service mesh provides distributed per-service proxies with telemetry and policies for east-west traffic; LBs are often centralized points for ingress/egress.

H3: How do I test my load balancer?

Run synthetic load tests covering peak patterns, simulate unhealthy backends, and perform chaos tests to validate failover and scaling.

H3: How do I monitor TLS certificate expiry?

Track certificate metadata via monitoring integrations and alert weeks or days before expiry; automate renewals using ACME or provider tools.

H3: How do I reduce alert noise for LB alerts?

Use grouping, aggregate thresholds with time windows, suppress during deployments, and tune thresholds to avoid transient noise.

H3: How do I implement sticky sessions securely?

Use signed cookies or tokens with short TTLs and prefer stateless session stores to avoid affinity-based overloads.

H3: How do I handle sudden traffic spikes?

Configure autoscaling for backends, have rate-limiting and load-shedding policies, and use CDN to absorb static traffic.

H3: How do I debug client reports of errors when LB metrics look fine?

Correlate client-side traces with LB logs, check TLS compatibility, and verify DNS routing and CDN caching.

H3: How do I set reasonable SLOs for LB latency?

Start from user experience and baseline measurements; set 95p/99p targets informed by business tolerance and iterative refinement.

H3: How do I manage multi-region traffic with LB?

Combine regional LBs with a global traffic manager using health checks and latency-based routing policies.

---

Conclusion

Load balancers are central to modern cloud architectures, affecting availability, latency, security, and operational practices. They enable safe deployments, traffic management, and are a critical observability point for SRE teams.

Next 7 days plan

• Day 1: Inventory current LBs, TLS certs, and health checks.
• Day 2: Ensure metrics and logs from LBs are collected centrally.
• Day 3: Define or review SLOs and error budgets for key endpoints.
• Day 4: Create runbooks for top 3 LB failure modes.
• Day 5: Automate LB config via IaC and add pre-deploy validation.
• Day 6: Run a small canary deployment and monitor LB signals.
• Day 7: Conduct a tabletop incident review and adjust alerts.

---

Appendix — load balancer Keyword Cluster (SEO)

• Primary keywords
• load balancer
• application load balancer
• network load balancer
• cloud load balancer
• ingress controller
• reverse proxy
• layer 4 load balancer
• layer 7 load balancer
• load balancing
• traffic distribution
• TLS termination
• session affinity

• weighted routing

• Related terminology

• health checks
• target group
• backend pool
• consistent hashing
• round robin
• least connections
• weighted least connections
• canary deployment
• blue green deployment
• service mesh
• Envoy proxy
• HAProxy
• NGINX ingress
• kube-proxy IPVS
• PgBouncer
• ProxySQL
• CDN edge caching
• DDoS protection
• WAF rules
• certificate rotation
• mutual TLS
• proxy protocol
• TLS handshake errors
• connection table exhaustion
• SYN flood mitigation
• global traffic manager
• anycast routing
• DNS TTL failover
• rate limiting
• load shedding
• observability for LB
• Prometheus exporters
• OpenTelemetry tracing
• request rate metrics
• latency percentiles
• error budget
• SLI SLO for LB
• burn rate alerts
• on-call runbook
• IaC for load balancer
• LB configuration drift
• autoscaling integration
• session stickiness cookie
• TLS offload vs re-encrypt
• edge routing patterns
• internal L4 proxy
• perimeter security
• ingress resource
• managed LB costs
• performance tuning
• connection draining
• readiness and liveness probes
• proxy-based retries
• traffic throttling
• health check frequency
• circuit breaker patterns
• debug dashboard panels
• synthetic transactions
• chaos engineering for LB
• multi-region active active
• failover test
• certificate management automation
• rate limit headers
• IP blacklisting
• network ACLs
• backend latency distribution
• request tracing headers
• x-forwarded-for handling
• signed cookies for affinity
• LB audit logs
• config validation tests
• deployment rollback strategy
• monitoring retention for incidents
• CDN vs LB role
• API gateway differences
• managed vs self-hosted LB
• performance baselining
• cost optimization for LB
• load balancer best practices
• load balancer tutorial
• enterprise LB architecture
• small team LB setup
• Kubernetes ingress tutorial
• serverless custom domain routing
• LB incident response checklist
• LB troubleshooting guide
• LB metrics to monitor
• LB alerts and suppression
• LB runbooks and playbooks
• LB security checklist
• LB integration map
• LB glossary terms
• LB implementation guide
• LB scenario examples
• LB common mistakes
• LB anti patterns
• LB operating model
• LB automation priorities
• LB observability pitfalls
• LB capacity planning
• LB load testing
• LB latency optimization
• LB configuration APIs
• LB third party tools
• LB logging best practices
• LB traceability techniques
• LB session management strategies
• LB global routing strategies
• LB regional failover planning
• LB performance tuning checklist
• LB canary deployment example

---