← 全部新闻
devops4 分钟阅读

Graceful shutdown: why your Node.js app is dropping requests on deploy

作者:Seaworthy · 2026年5月14日

Aerial photograph of an airport runway at daytime, shot from above

Photo by Domaintechnik Ledl.net on Unsplash

Every time you deploy, Kubernetes kills your old pod. The question is whether it kills it cleanly or just pulls the rug out from under whoever is mid-request.

Most Node.js apps I review get this wrong. Not because the developer didn't care, but because no one told them it was a problem.

What actually happens between SIGTERM and container death

When Kubernetes decides a pod needs to go, it does two things at once: it sends SIGTERM to PID 1 inside the container, and it removes the pod from the Service endpoints so no new traffic gets routed there. The removal propagates through kube-proxy and any load balancer, but that takes a few seconds. During those seconds, requests are still arriving.

After terminationGracePeriodSeconds (default: 30s), Kubernetes sends SIGKILL. No negotiation. Whatever is running dies.

So the window you have is: SIGTERM arrives, you stop accepting new connections, you finish the ones in flight, you exit. Simple in theory.

Why process.exit(0) in the signal handler is wrong

This is the pattern I see constantly:

process.on('SIGTERM', () => {
  process.exit(0);
});

That's not graceful shutdown. That's just a slightly more intentional version of SIGKILL. Any request that arrived before the signal and hasn't finished yet gets dropped. The client gets a connection reset. You lose data, you break user flows, and if you're writing to a database mid-transaction, you corrupt state.

Some apps don't even have the signal handler. Node.js will exit on SIGTERM by default, which produces the same result.

Stopping the accept loop while draining existing connections

The right move is to call server.close(). This stops the HTTP server from accepting new connections while letting existing ones finish. Then you exit once the server has drained.

Here's a complete, production-ready implementation:

import http from 'http';
import { app } from './app'; // your Express/Fastify/Koa app
import { db } from './db'; // your database client
import { flushJobs } from './jobs'; // any background job queue

const server = http.createServer(app);
const PORT = process.env.PORT ?? 3000;

server.listen(PORT, () => {
  console.log(`Listening on ${PORT}`);
});

async function shutdown(signal: string) {
  console.log(`Received ${signal}. Shutting down gracefully.`);

  // Stop accepting new connections
  server.close(async (err) => {
    if (err) {
      console.error('Error during server close:', err);
      process.exit(1);
    }

    try {
      // Flush any pending background jobs before closing
      await flushJobs();

      // Close database connections cleanly
      await db.destroy();

      console.log('Shutdown complete.');
      process.exit(0);
    } catch (cleanupErr) {
      console.error('Error during cleanup:', cleanupErr);
      process.exit(1);
    }
  });

  // Safety net: force exit if cleanup takes too long
  setTimeout(() => {
    console.error('Shutdown timed out. Forcing exit.');
    process.exit(1);
  }, 25_000).unref();
}

process.on('SIGTERM', () => shutdown('SIGTERM'));
process.on('SIGINT', () => shutdown('SIGINT'));

Notice the 25-second forced timeout. Your terminationGracePeriodSeconds is 30s by default, so this gives you 25 seconds to clean up before Kubernetes sends SIGKILL. If you let cleanup hang indefinitely, you'll eventually hit SIGKILL anyway, with no log and no visibility into why.

Database teardown and background job flushing

The order matters. Stop serving traffic first, then flush jobs, then close the database. Do it the other way around and you'll call db.query() after the pool is closed, which throws, which may also prevent process exit.

For background jobs, "flushing" means different things depending on your queue. If you're using BullMQ, you want to call worker.close() and wait for the current job to finish. If you're batching events to an analytics sink, you want to force a flush of the in-memory buffer. Whatever it is, it needs to complete before db.destroy().

For long-running database transactions, the honest answer is that graceful shutdown can't help you if you're holding a 20-second transaction open. Design transactions to be short. Graceful shutdown handles the request layer; transaction design is a separate problem.

terminationGracePeriodSeconds and the preStop hook

There's a race condition in Kubernetes that trips up even experienced engineers. When SIGTERM arrives and your app stops accepting connections, kube-proxy on other nodes might not have removed your endpoint yet. Requests can still arrive for a second or two after SIGTERM.

The fix is a preStop lifecycle hook that sleeps briefly before the signal is sent:

lifecycle:
  preStop:
    exec:
      command: ["/bin/sh", "-c", "sleep 5"]

This delays SIGTERM by 5 seconds, giving kube-proxy time to propagate the endpoint removal. Your app is still running and accepting requests during that sleep. After it ends, SIGTERM arrives and you drain. Set terminationGracePeriodSeconds to account for the sleep time plus your expected drain time:

spec:
  terminationGracePeriodSeconds: 35

5s sleep + 25s drain + 5s buffer = 35s. Adjust based on your actual p99 request duration.

I'll be honest: this two-part solution (preStop sleep plus graceful drain) took me an embarrassingly long time to land on. The Kubernetes docs mention it, but the connection between "endpoint propagation latency" and "dropped requests on deploy" isn't spelled out clearly. You usually discover it the hard way, watching 502s tick up in your APM during a rollout.

Get this right and deploys become non-events. No 502 spikes, no angry Slack messages, no rollbacks. It's not glamorous work, but it's the kind of thing that separates a reliable service from one that quietly drops 0.1% of requests on every release.

Seaworthy flags missing SIGTERM handlers and bare process.exit() calls in signal handlers automatically.

本文由 AI 生成,总结了公开可用的来源。

来自 Seaworthy 团队

Seaworthy 会扫描你的代码仓库,发现类似本文提到的各种问题。

安全漏洞、泄露的密钥和配置错误——在部署前捕获。免费运行,无需账户。

$ npx seaworthycode