Why a missing health check endpoint breaks zero-downtime deploys

You deploy a new version. Watchtower or your CI pipeline pulls the image, spins up a new container, and starts routing traffic to it. Thirty seconds later your error rate spikes, half your users get 502s, and the on-call channel lights up. You didn't ship a bug. You shipped a container that hadn't finished booting yet, and nothing told the orchestrator to wait.

This is the single most common cause of "it worked in staging" deploy failures, and the fix is about twenty lines of code that most vibe-coded projects never write.

How orchestrators actually decide a container is ready

Every deployment tool that claims "zero downtime" is lying unless something is checking readiness. Here's what each one actually does:

A load balancer (nginx, HAProxy, an AWS ALB) polls a configured path, usually /health, on a timer. Get a 200, the backend stays in the pool. Get a non-200, a timeout, or a connection refused, it's pulled out of rotation. No health check configured means the load balancer treats "the port accepted a TCP connection" as ready, which happens well before your app has finished loading config, warming caches, or opening a DB pool.

Kubernetes uses two separate probes for two separate questions, and conflating them is its own class of outage. The readiness probe answers "should this pod receive traffic right now" and gates entry into the Service's endpoint list. The liveness probe answers "is this process stuck" and, if it fails repeatedly, gets the pod killed and restarted. Skip the readiness probe and Kubernetes assumes ready the instant the container starts, so requests hit a pod that's still running migrations or hasn't connected to Redis yet.

Docker Swarm has a HEALTHCHECK instruction in the Dockerfile (or a healthcheck: block in compose). Without it, Swarm considers a container healthy the moment its process starts, full stop. A rolling update will happily replace every old container with new ones that are still initializing, one after another, because nothing is holding it back.

Watchtower-style auto-updaters, the kind that poll a registry and swap containers on a new digest, are the worst offender because they're usually paired with nothing else. Watchtower pulls the new image, stops the old container, and starts the new one. It does not wait for the new one to be ready unless you've wired a health check into the compose file and your reverse proxy respects it. Left unconfigured, every auto-update is a guaranteed few seconds (sometimes much longer, if there's a slow migration or a cold cache) of dropped or 502'd requests.

The two failure modes without a health endpoint

Not ready yet. The new container is up but the app hasn't finished startup, maybe it's running migrations, warming an in-memory cache, or waiting on a downstream dependency. Traffic arrives anyway because the orchestrator has no way to distinguish "process started" from "process ready."

Silently wedged. A deadlocked event loop, a DB connection pool that's exhausted and never recovers, a background job that's spinning and starving the request handler. The process is still running, so a liveness check (or the TCP-connect check a naive setup falls back to) says it's fine. It isn't. Without a real liveness/readiness signal, that container sits in the traffic pool serving timeouts until someone notices manually.

Both failures look identical from the outside: users get errors, dashboards show latency spikes, and the postmortem says "the deploy caused an outage." The actual cause is that nothing was checking.

A health check that always returns 200 is worse than no health check

This is the trap. Teams add /health because a linter or a template told them to, and it looks like this:

app.get('/health', (req, res) => {
  res.status(200).send('ok');
});

This satisfies the orchestrator's config requirement and does nothing useful. It returns 200 whether the database is reachable or not, whether migrations have run or not, whether the process is one exception away from crashing. You've added the box-ticking without the safety net. A container that can't reach its database will report itself healthy right up until the first real request fails.

A health check earns its name by testing something that can actually fail:

app.get('/health', async (req, res) => {
  try {
    await db.query('SELECT 1');
    res.status(200).json({ status: 'ok' });
  } catch (err) {
    res.status(503).json({ status: 'error', detail: 'database unreachable' });
  }
});

That single query is the difference between an orchestrator that can protect you and one that's cosplaying protection. If the DB connection pool is exhausted, the pool hasn't finished initializing, or the network path to the database is down, this returns 503 and the container gets pulled from rotation instead of eating live traffic.

A couple of things worth being deliberate about:

Set a short timeout on the check itself (a few hundred milliseconds is plenty for SELECT 1), so a slow database doesn't turn your health check into another thing that hangs. And don't check everything downstream, an optional third-party API being flaky shouldn't take your whole service out of rotation if it isn't on the critical path for serving requests. Check what actually has to work for a request to succeed, nothing more.

For Kubernetes specifically, wire this into a readinessProbe, and keep a separate, cheaper livenessProbe (one that doesn't hit the DB) so a slow database doesn't get your pod restarted in a loop on top of everything else.

What to wire it into

The endpoint alone does nothing if nothing polls it. Minimum viable setup:

• Load balancer or reverse proxy: point its health check config at the route.
• Kubernetes: add readinessProbe (checks the dependency) and livenessProbe (checks the process is alive, nothing more) to the pod spec.
• Docker Swarm / Compose: add a HEALTHCHECK instruction with --start-period set long enough to cover your actual boot time, so a slow-starting app isn't marked unhealthy before it's had a chance.
• Watchtower or any pull-and-swap updater: pair it with a HEALTHCHECK in the image and a proxy that respects container health state before routing. Watchtower itself doesn't gate traffic, your proxy has to.

None of this is exotic. It's the difference between a deploy pipeline that fails safely and one that fails onto your users.

Seaworthy's static analysis flags services with no /health or equivalent readiness endpoint defined, since it's the single check most likely to turn a routine deploy into an incident.