Scaling Out

The "Scale Cube" (from The Art of Scalability by Abbott & Fisher) provides a mental model for three dimensions of scaling:

X-axis scaling (this module) is the prerequisite for everything else. If your application can't run as multiple identical clones, you can't decompose it into microservices or shard its data. The key requirement: your application must be stateless.

Horizontal scaling only works if your application servers are stateless — meaning no server holds any user-specific data in local memory. If a load balancer sends Request 1 from User A to Server 1 and Request 2 to Server 3, both servers must be able to serve User A identically.

What "state" do applications commonly hold in memory?

• Session data: Login status, shopping cart, form wizard progress
• File uploads: Temporary files saved to local disk
• In-process caches: Application-level data caches (handled in Module 3)
• WebSocket connections: Long-lived connections bound to a specific server
• Background job state: In-progress job tracking stored in memory

Each of these must be externalized to a shared store to enable stateless scaling.

Session Management Strategies

The most common stateful element is the user session. Here are three strategies for externalizing it, each with distinct tradeoffs:

Running a fixed number of servers wastes money during low traffic and underserves during peaks. Auto-scaling dynamically adjusts the number of instances based on demand.

Scaling Metrics

What metric should trigger scaling? The answer depends on your bottleneck:

Scaling Policies

• Reactive scaling responds to current metrics: "If CPU > 70% for 3 minutes, add 2 instances." Fast to implement but always lags behind demand.
• Predictive scaling uses historical patterns: "Every Monday at 9 AM, traffic increases 5x, so pre-scale at 8:55 AM." AWS Auto Scaling and GCP Autoscaler support this natively.
• Scheduled scaling uses cron-like rules: "Set min instances to 20 during business hours, 5 at night." Useful for known traffic patterns (e-commerce sales events, streaming premieres).

When you add or remove servers dynamically (auto-scaling), how does the rest of the system know where to send traffic? Hardcoding IP addresses doesn't work when instances are ephemeral. Service Discovery solves this by maintaining a registry of available service instances.

In a fleet of servers, instances fail constantly — hardware dies, processes crash, deployments roll out. The system must detect and react to failures automatically, without human intervention.

Health Check Types

• Liveness probe: "Is the process alive?" A simple HTTP 200 response. If this fails, restart the instance. Example: GET /healthz → 200 OK.
• Readiness probe: "Can this instance serve traffic?" Checks database connectivity, cache availability, and warm-up status. If this fails, don't send traffic to this instance, but don't restart it.
• Startup probe: "Has the app finished starting?" Important for JVM applications with 30-60 second startup times. Prevents the liveness probe from killing the app before it's ready.

Graceful Shutdown

When an instance is removed (scaling in, deploying a new version), it must shut down gracefully:

1. Stop accepting new connections — deregister from the load balancer.
2. Drain in-flight requests — wait for active requests to complete (with a timeout, e.g., 30 seconds).
3. Close resources — close database connections, flush logs, commit offsets.
4. Exit — terminate the process with exit code 0.

In Kubernetes, this is orchestrated by the SIGTERM signal followed by a configurable terminationGracePeriodSeconds (default: 30s). If the pod hasn't exited by then, it receives SIGKILL.

Choosing the right LB algorithm determines how evenly traffic is distributed. No single algorithm is best for all workloads.

Horizontal scaling changes how you deploy. You can no longer "stop the server, update, restart." You need zero-downtime deployment strategies:

File Uploads → Object Storage

If users upload files (images, documents, videos), storing them on the local filesystem breaks with horizontal scaling — Server 1 has the file, but Server 2 doesn't. The solution: object storage (S3, GCS, Azure Blob). Upload directly to S3 using pre-signed URLs — the app server never touches the file, reducing CPU and memory load.

WebSocket Scaling

WebSocket connections are long-lived and stateful by nature — a user connects to Server 1, and all messages must flow through Server 1. This contradicts stateless scaling. Solutions:

• Redis Pub/Sub for fan-out: Each server subscribes to relevant Redis channels. When a message arrives for a user on Server 1, it's published to Redis, and Server 1's subscription delivers it. All servers can receive messages for any user.
• Sticky connections with session awareness: Use L7 LB cookie-based routing to keep WebSocket connections on the same server, combined with a Redis-backed mapping of user → server for targeted message delivery.

• The Art of Scalability by Martin Abbott & Michael Fisher — The Scale Cube (X/Y/Z axes) defines the three dimensions of scaling. (Addison-Wesley, 2015)
• Designing Data-Intensive Applications by Martin Kleppmann — Chapter 6 (Partitioning) and Chapter 8 (Trouble with Distributed Systems). (O'Reilly, 2017)
• The Netflix Simian Army — Netflix Tech Blog — How Chaos Monkey, Chaos Kong, and Latency Monkey validate resilience.
• How Discord Stores Trillions of Messages — Scaling real-time systems to millions of concurrent connections.
• Kubernetes HPA Documentation — Official guide to Horizontal Pod Autoscaler, including custom metrics.
• Canary Release — Martin Fowler — The original description of canary deployment strategy.
• Stateless Auth for Stateful Minds — Auth0 — JWT-based session management patterns and tradeoffs.
• Release It! by Michael Nygard — Stability patterns including circuit breakers, bulkheads, and timeouts for production scaling. (Pragmatic Bookshelf, 2018)