The Health Check That Caused the Outage

Your database dropped out for a few minutes. Your health check turned that into a full outage.

That's not a hypothetical situation I made up for effect. It happened to us twice, in two different shapes. Here's the first.

We had a health check that reached DocumentDB. One day a Security Group rollout didn't go to plan, and the app could no longer reach DocumentDB. It sat behind maybe a tenth of our endpoints, so the damage should've been small.

It wasn't. The processes never crashed, the app was alive on every instance. But the health check couldn't reach DocumentDB, so it flagged every instance as unhealthy at the same moment, and the platform started cycling the whole HTTP tier out, faster than the replacements could come back. The workers kept running with nothing to do, because the thing that fed them had gone dark. Took us a while to trace it back to one Security Group change. A dependency that maybe a tenth of the product needed had taken down all of it.

Putting that database check in the health endpoint feels responsible, and that's why so many teams do it.

The assumption every orchestrator makes

Start with what a health check is for. The load balancer, the autoscaling group, the Kubernetes control plane: all of them run the same loop. Find the unhealthy instance, take it out of rotation, put a healthy one in. Route around the broken thing.

That loop has a hidden assumption baked in. It assumes failures are independent. And it works because when one instance breaks, the others are fine, so there's somewhere to send traffic and something to replace the dead node with.

A deep health check breaks that assumption. The moment your check queries a dependency that every instance shares, you've tied the health of all of them to one external thing. When that thing goes down, every instance fails its check in the same second. The machinery built to route around one broken instance now has nothing healthy to route to. It was never designed for correlated failure, and you created correlated failure on purpose.

That's the core of it. The rest of this post is what it looks like on EC2, ECS and EKS, and the way out, which turns out to be one principle applied a little differently each time.

Liveness vs readiness: the same mistake, two outcomes

Liveness and readiness are the two jobs a health check can have: is the process alive, and should it get traffic right now. Kubernetes splits them into separate probes; most other setups blur them into one.

Where you wire the deep check decides how badly it hurts.

On readiness, all instances go NotReady together when the dependency blips. The load balancer sees zero healthy targets. Now even requests that never touched the database get nothing back, because there's no instance left in rotation to serve them. A local problem becomes a total one.

On liveness, it's worse. The orchestrator doesn't only stop sending traffic. It decides the app is dead and starts killing and restarting processes, because a dependency is slow. So at the exact moment the system is fragile, you add cold starts, fresh connection storms, and more pressure on the dependency that was already struggling. You're feeding the fire you were trying to put out.

Liveness answers one question: is this process alive and able to serve a trivial request. It doesn't touch business dependencies. If the process is alive, liveness keeps succeeding and the orchestrator leaves it running. Whether traffic is routed to it is a separate, readiness concern. Endpoints that actually need the database return 503 from the app's error handler, per request, when the database is down. That 503 comes from your code, not from the infrastructure pulling the whole instance out of service.

The same trap on EC2 + ASG + ALB

Most writing on this assumes Kubernetes. The trap is the same on plain EC2, slower and more expensive, and both of our incidents ran there: behind an ALB, with an Auto Scaling Group.

The setup is ordinary. The ALB target group runs a health check against an endpoint. The ASG has health check type = ELB, so it replaces any instance the load balancer marks unhealthy. Reasonable, until the checked endpoint reaches a shared dependency. Then one downstream failure fails every target at once, the ASG marks the whole fleet unhealthy, and it does what you configured: terminate and replace.

Here's where it turns into a loop that's hard to break from the inside. When the ASG launches a replacement, CodeDeploy runs the deployment lifecycle on that fresh instance before it enters service, validation hook included. If that validation hook also reaches the dependency, it fails too, because the dependency is still down. So the replacement never passes validation and never joins the fleet. The old instances are cycled out by the health check, and the new ones can't get in past the deploy gate. The fleet can't even rebuild itself with the artifact it already has, because the thing failing is the dependency check, not the code. That's the rotation the DocumentDB outage put us in. You don't recover until the dependency does.

An ASG doesn't understand any of this. It reasons about a single number: desired capacity minus current capacity. It sees a missing instance and launches one, with no idea that every instance is failing for the same reason. So it keeps throwing fresh machines at a wall, each one hitting it in the same spot.

One AWS detail matters here, because the earlier framing is too simple for the ALB specifically. An ALB doesn't black-hole traffic the moment every target is unhealthy. It fails open: with no healthy target in a group, it routes to all of them anyway, on the assumption that the check is the thing at fault. So on EC2 the lost capacity isn't the ALB refusing to route. It's the ASG terminating the instances the ALB flagged, and the failed replacements never coming up behind them.

The shallow version is undramatic. The ALB check hits a lightweight endpoint that touches no external dependencies, the ASG replaces an instance only when the shallow check says that instance itself is unhealthy, and the maintenance policy launches a replacement before terminating the old instance, so capacity never dips.

# ALB target group: shallow check, nothing downstream behind it
resource "aws_lb_target_group" "app" {
  health_check {
    path    = "/health"   # process-alive only, never the database
    matcher = "200"
  }
  deregistration_delay = 30   # drain in-flight requests on the way out
}

# ASG: replace when the shallow check fails, launch before terminating
resource "aws_autoscaling_group" "app" {
  health_check_type = "ELB"

  instance_maintenance_policy {
    min_healthy_percentage = 100
    max_healthy_percentage = 110
  }
}

None of it helps if the endpoint behind path = "/health" reaches a shared dependency.

The second incident: when more instances made it worse

Underneath, this outage and the DocumentDB one were the same: a connectivity problem that automation escalated into a fleet-wide one, both rooted in infra config. What made this one harder to climb out of was an autoscaler that took the cascade and pulled it tighter. It's the same machinery as the earlier replacement loop, pointed the other way: instead of swapping instances out, it kept piling new ones on.

We'd been scaling up to chew through a big backlog, and the capacity got ahead of what the database could take. As I remember it, enough workers came up to use up the database's connection limit, and RDS stopped accepting new connections. From there it was a feedback loop.

You might expect a stalled database to leave the app idle, blocked on I/O, with the CPU relaxed. It went the other way, and the scaling rule fired on CPU. I was on the sidelines for this one, so I'm reconstructing the compute side, but the shape is familiar: the box wasn't waiting quietly, it was working hard at failing. Workers piled up blocked on connections that never came, every failed request threw and was logged, and a log-shipping agent churned through the flood. None of it is useful work, but all of it burns CPU, and to a scaling rule a pinned CPU looks like demand.

So it scaled out, the worst thing it could do. Every new instance opened its own connections on boot, so the moment a slot freed up a rising instance grabbed it before anything stable could. The fleet climbed to its ceiling and kept rotating, and none of it helped, because the bottleneck was the database, not the compute.

The fix had to go where the bottleneck was, at the dependency: more connections, a bigger instance, the usual capacity work. A connection proxy would've softened it too, by pooling connections instead of letting every instance open its own, but softening is not removing. A proxy still fronts a finite database, and not every dependency has one. The cleaner answer is to stop generating the pile-on.

What stuck with me was the shape of it. More instances is the standard answer to load. Here, more instances kicked us in the face. The autoscaler assumed every new instance would help. But every new instance made the starvation worse.

ECS sits in the middle

ECS lands between the manual EC2 setup and full Kubernetes, and the trap shows up there too, under different names.

ECS gives you two health signals. The container health check, the HEALTHCHECK in your task definition, behaves like liveness: fail it and ECS replaces the task. The target group health check, run by the ALB, behaves like readiness: fail it and the task is deregistered from traffic, and replaced. Put a dependency check in either one and you get the same cascade as everywhere else, with ECS rescheduling the whole set of tasks against a dependency that's still down.

The one ECS-specific knob here is healthCheckGracePeriodSeconds, which gives a freshly started task a window before the ALB health check can mark it unhealthy. Notice what it does and doesn't cover: it buys time for a slow boot, nothing more. It says nothing about whether a downstream is healthy. Those are two different problems, and ECS has no separate signal for the second one, so the workaround later in this post applies here too.

Kubernetes splits the one signal into three

I'm still getting to know Kubernetes. I come from the EC2 and ASG side, where you wire rotation and replacement together by hand. What struck me while learning EKS is that the platform separates three questions EC2 forces you to answer with a single signal: is this alive, should it get traffic, is it still starting.

Each question maps to a different action. Kubernetes restarts a container that fails liveness. It holds traffic off a pod that hasn't passed readiness. It waits through a slow boot with the startup probe instead of killing the pod for being slow. A rolling update never sends traffic to a pod until it reports ready.

It's easy to overstate the startup probe. It covers slow startup and nothing else. None of the three probes was built to express dependency health. Each answers one thing about the instance itself: liveness, the process is alive; readiness, this pod can take traffic; startup, the boot has finished. People do wire a downstream check into readiness, and sometimes that's right, but it's something you add on top. What the platform gives you by default is those three concerns kept separate. That's the real difference from EC2, where there's one signal, so everything gets crammed into it, which is also why the deep-check mistake is so easy to make there.

Readiness is about useful work

Readiness has a real job, and it's not checking dependencies. It gates traffic during rolling deploys and graceful drain. A pod starts, the liveness check returns 200 right away because the process is up, but the caches are cold and the connection pools are empty, so the first requests would be slow or fail. Readiness holds traffic off until the pod is warmed up, and during shutdown it lets you drain in-flight requests before the process exits. It's about this instance's own state, not the health of anything downstream.

Which is why the common correction misses. "Don't check the database in liveness, check it in readiness instead" just moves the deep check. On a shared dependency the correlated failure comes along for the ride: the dependency blips, every instance reports not-ready at once, and the load balancer is back to zero healthy targets, with all the operational mess that brings.

There's at least one shape where a readiness check on a dependency is the right call. Picture a small, single-purpose service: it takes a message and writes it to a queue, and that's all it does. If the queue is gone, the service has no useful work left. It can do nothing except return 503, and every one of those failed requests is another connection thrown at a dependency that's trying to recover. A healthy-but-useless instance hammering an unhealthy dependency can be the very thing that keeps it from coming back. Here, dropping the instance from rotation simply helps. There's no partial availability to protect, because every request needs the one thing that's down, and pulling out of routing takes pressure off the dependency while it recovers.

The distinction is footprint. A readiness check on a dependency is right when that dependency is the service's whole reason to exist. It becomes the outage from the start of this post the moment the dependency sits behind only part of a larger app and gets treated as a prerequisite for all of it. So readiness answers "can I do useful work," not "is my shared downstream alive."

When the platform gives you one signal

What do you do when you can't express "alive but degraded" at the instance level, because there's one health signal and it's binary. Maybe you're on EC2 with a single ALB check, maybe your platform doesn't separate the probes the way EKS does. The move is to push the dependency awareness down one level, from the instance to the request.

Keep /health shallow. It returns 200 if the process can serve a trivial request, and that signal alone decides rotation and replacement. Then let each endpoint decide for itself. An endpoint that needs the database tries the database and returns 503 if it's down. An endpoint that doesn't need the database keeps returning 200 and keeps serving. The blip degrades only the endpoints that depend on the broken thing. The rest stay up.

This has a name worth using: partial availability. The goal is to keep serving the requests that don't need the dependency, not to pretend it's healthy. If 20% of your requests depend on the failed component, fail those 20% and keep the other 80% working. A shallow check plus per-request errors gets you that. A deep check gets you zero percent, because it takes the whole fleet down over a dependency that most requests never touch.

One honest limit of the per-request approach: it still reaches for the dependency on every request. Each call tries, waits, and fails, so a struggling dependency keeps getting poked while it recovers. The usual tool for cutting that off is the circuit breaker: past a failure threshold it stops trying, returns the error at once, and probes in the background until the dependency is back. That's a post in itself, so I'll leave it here as a name to look up.

The real fix is keeping shared dependencies out of the signal that drives replacement and routing. A shallow /health is usually the simplest way to do that. There's a blunter fallback when you can't change the check itself: set health check type = EC2, so the ASG replaces only on instance-level failure and a failing ALB check can no longer trigger termination. It breaks the churn loop, but treat it as a last resort. The ELB check is what catches a process that's running but wedged: a deadlocked app, a server returning stale 500s, something alive but no longer answering. Drop to type = EC2 and you lose that, and have to rebuild it with a watchdog of your own.

Correlated failure is the pattern

Most of us design the happy path well. The question that gets asked too rarely is "what happens when a dependency gets slow or goes away." A health check is one of the few places where that question has a default answer already baked in, and the default that feels most responsible, prove the whole stack works on every check, is the one that converts a small dependency blip into a system-wide outage.

The pattern runs through every layer. Health checks, load balancing, auto-replacement, auto-scaling: all of it assumes failures are roughly independent, and that adding or replacing capacity helps. The machinery routes around the broken instance because the other instances are fine. Tie every instance to the same shared dependency and that assumption is gone. One slow downstream takes the whole fleet together, and the system built to save you either has nothing left to route to, or it scales straight into the bottleneck and makes recovery impossible.

The checks earn their keep by staying simple. Liveness: is this process alive. Readiness: can this instance do useful work yet. Neither asks whether a shared downstream is healthy, because that's a per-request question. Answer it per request, and a broken dependency costs you the requests that needed it. Answer it at the instance level, and it costs you everything.

Get the distinction right and the same blip stays contained: only the endpoints that needed the broken dependency fail, the rest keep serving. No fleet-wide rotation, no autoscaler digging the hole deeper, no recovery fought by your own automation. The machinery built to save you finally does it.

Background workers run into the same correlated-failure pattern from another direction: an HTTP health endpoint can happily report a live process while the work loop itself is dead. That's its own post.