Your backup cron job has been failing silently for three weeks

The backup job runs at 3am. It has run at 3am every night for the last eight months. Nobody has looked at its output in eight months either, because it's never once been the reason anyone got paged. That's exactly the setup where a silent failure lives the longest.

Here's the version of this script most people write:

try {
  await runBackup()
} catch (err) {
  console.error('Backup failed:', err)
}

This looks like error handling. It is error handling, in the sense that it stops the process from crashing. But console.error in a cron job that nobody tails is a message shouted into an empty room. The job "succeeds" from the scheduler's point of view every single night, whether the backup actually completed or not, because the process exited without an unhandled exception. Three weeks later someone needs to restore from a backup and finds out the last one that actually worked was from three weeks ago.

The gap between "ran" and "worked"

A cron scheduler, a Kubernetes CronJob, or a systemd timer all report the same thing: did the process start and exit cleanly. None of them know whether the backup file is non-empty, whether the upload to S3 actually completed, or whether the database dump silently truncated because a connection dropped halfway through. "The job ran" and "the job worked" are different claims, and most monitoring setups for scheduled tasks only check the first one.

A dead man's switch closes that gap

The pattern is simple: instead of only alerting when something fails, you alert when you stop hearing that something succeeded. A cron job pings a monitoring endpoint on successful completion. If that ping doesn't arrive within the expected window, the monitoring service pages you, whether the job crashed, hung, got skipped because the container never started, or the whole box lost power.

async function runBackupWithHeartbeat() {
  await runBackup()
  await fetch('https://hc-ping.com/your-unique-check-id', {
    signal: AbortSignal.timeout(5000),
  })
}

runBackupWithHeartbeat().catch((err) => {
  console.error('Backup failed, no heartbeat sent:', err)
  process.exitCode = 1
})

Services like Healthchecks.io, Cronitor, and Better Uptime all offer this as a first-class feature, usually on a free tier for a handful of checks. You give them an expected schedule, they give you a URL to ping, and they own the job of noticing when the ping stops arriving. That's the whole integration. No agent to install, no library beyond whatever you already use to make an HTTP request.

Alert on absence, not just on error

The reason this matters more than a better try/catch is that most of the ways a scheduled job actually fails don't produce an error at all. The container image never got redeployed after a config change and the job silently stopped being scheduled. A typo in a cron expression means it runs on the 31st of months that don't have one. The process OOMs and gets killed before it reaches the catch block. In every one of these cases there's no exception for your logging to capture, because nothing in your code ever ran to fail.

A dead man's switch doesn't care why the ping didn't arrive. It only cares that it didn't. That's a much smaller surface to get wrong than trying to anticipate every way a job can go quiet.

What this doesn't replace

A heartbeat tells you the job ran and exited without throwing. It doesn't verify the backup is restorable, only that the process that's supposed to produce one finished. Pair it with an occasional actual restore test, since a corrupted backup that uploads successfully will ping the switch just fine and still be useless the day you need it.

Seaworthy checks for scheduled jobs and backup scripts with no monitoring or alerting configured, since a job that fails quietly for weeks is a much worse outage than one that fails loudly on day one.