More Replicas, the Same Database

The application is slow, so the autoscaler adds replicas.

Each new replica opens its normal database connection pool. The database was already waiting on locks. Now it has more callers waiting on the same locks.

CPU on the application instances improves. Checkout latency does not.

Nothing in that sequence is especially exotic. We scaled the part that could multiply, without checking the capacity of the part all those copies still share. I find that a more useful starting point than the usual discussion of which load-balancing algorithm looks fairest.

The unit of scaling includes its dependencies

An application replica is not just a process with some CPU. It is also a database pool, a set of cache connections, outbound calls, retry behavior, and startup work.

Consider a simplified connection budget:

database limit:                  200
reserved for administration:      20
workers and other services:        40
available to the web application: 140

8 replicas x 15 pool connections: 120
12 replicas x 15 connections:     180

These numbers are illustrative. A pool maximum does not mean every process immediately opens every connection, but it is a capacity promise we need to bound. A rolling deployment can temporarily run both old and new replicas. The maximum deployment overlap belongs in the budget too.

I would cap replica count and per-process concurrency together. A connection pooler may help, but it cannot make a transaction release its locks sooner or make a slow query cheaper.

Queuing moves; it does not disappear

If the application accepts work faster than it can finish, requests wait somewhere: at the proxy, inside the process, in the pool, or in the database.

An unlimited internal queue can make an overloaded service look healthy until requests exceed every client timeout. By then the service may still be doing work for callers that already gave up.

Explicit admission limits give the system a more useful response. Bound concurrency, reject excess work with a deliberate status, and decide whether a retry can safely occur. Do not spend thirty seconds doing an operation whose result the caller is no longer waiting for.

Cancellation matters, but cancelling a browser request does not automatically cancel the database operation. Deadline propagation needs to reach the layers that support it, with transaction cleanup handled deliberately.

I would prefer a small number of quick, understandable overload responses to a large number of ambiguous timeouts. That choice is less attractive on a throughput chart and more attractive during an incident.

Retries can manufacture the next spike

A browser retries. The gateway retries. The application retries its upstream call. Three attempts at each layer can multiply rather than merely add.

If the original work is slow because a shared dependency is saturated, retrying immediately adds more work to that dependency. The request deadline may also restart at each hop, extending the actual time spent far beyond what the caller expects.

Assign retry ownership. Use a bounded budget, backoff, jitter, and idempotency for writes. Honor useful rate-limit guidance such as Retry-After rather than asking every client to guess its own recovery rhythm.

Retries are appropriate when another attempt has a reasonable chance of finding different conditions. They are not a substitute for an overload policy.

Stateless does not mean there is no state

The practical meaning is narrower: another instance can handle the next request without relying on unique state in the previous instance's memory.

Sessions may live in a shared store. Uploaded files belong in durable object storage. Order state belongs in the database. Any replica can reconstruct what it needs.

Local caches are still possible. So are in-memory counters and background bookkeeping, provided they are explicitly disposable and do not claim global correctness.

Sticky sessions can keep a client on one instance, which may be useful for a specific workload. They do not make that instance durable. When it disappears, the unique state disappears too unless a recovery path exists.

I would describe required affinity as a constraint rather than advertise a system as stateless while quietly depending on the load balancer to remember each user.

Health checks are routing policy

A liveness check asks whether the process can continue. A readiness check asks whether it should receive new work. A synthetic business check asks whether a useful operation succeeds. They do not necessarily belong in the same endpoint.

Suppose every instance's readiness check requires Redis to respond. Redis has a shared outage, so every instance becomes unready. Removing all instances may be correct for a session-only application that cannot authenticate anyone. It may be unnecessary for public product pages with a database fallback.

Restarting all processes because a shared dependency is slow can also worsen recovery by creating connection and cache-warming bursts.

There is a specific AWS detail worth knowing: ALB fails open when a target group has only unhealthy registered targets. It can route to those targets rather than treating health failure as a guarantee of no traffic.

This is not a recommendation to return false health results. It is a reminder to understand the platform's actual response and keep application safeguards in place. The health check does not become a security boundary just because the dashboard colors it red.

Scaling needs time to become useful

A new instance may need to pull an image, initialize connections, load configuration, and warm required state before becoming ready. Autoscaling follows observations with a delay; it cannot instantly absorb a sudden spike.

Keep enough baseline capacity for the workload that cannot wait, and choose signals that reflect the bottleneck. CPU can help for compute-heavy traffic. Queue age may better describe workers. Pool wait time or active-request concurrency may reveal pressure that average CPU hides.

Avoid a warmup procedure that makes every new replica perform the same expensive full-cache refresh. The capacity meant to help the database can briefly become its largest customer.

Scale-in is equally important. A quiet instance may still hold a long request or many WebSocket connections. Removing it is a lifecycle event, not just subtracting one from a counter.

Draining coordinates several timers

The load balancer has a deregistration delay. The process has a shutdown grace period. Requests have deadlines. Connections have idle timeouts. These settings need to agree on how ongoing work ends.

Stop admitting new requests, allow existing work a bounded interval, then shut down. If the platform kills the task before the drain interval finishes, the load balancer's generous delay does not help.

For WebSockets, draining may require application messages and client reconnection. A reconnected client can land on another replica; events and subscriptions need shared or reconstructible state. The replacement must not assume the old connection's in-memory history is available.

A slow deployment should not require every user to discover the reconnection policy through a frozen screen.

Test a removal, not only an addition

A useful scaling exercise removes an instance while traffic is active. It observes errors, latency, duplicate writes, session continuity, and long-lived connection recovery.

Test mixed versions too. Schema compatibility and old frontend bundles remain relevant while the load balancer serves both generations. Retain content-hashed assets long enough for active clients to finish or recover cleanly.

These tests make capacity a statement about the whole service rather than a count of containers.

I am comfortable with modest scale when the limits are known. I am less comfortable with an autoscaling policy whose only promise is that more processes will appear whenever something becomes slow.

The shared database has not read that promise. It will keep accepting only the work it can actually perform.

Next, the upcoming article on production observability looks at how to see these limits before an error report becomes the first measurement.

When your replica count grows, which downstream quantity grows with it? I'd start a scaling review with that number, before choosing another autoscaling signal.