Circuit breakers: stop one slow dependency from taking down your whole app

Your payment service goes down at 2am. That's bad, but survivable: checkout is broken, everything else still works. Now imagine your payment service goes slow instead. Requests pile up waiting for a response. Your thread pool fills. Your connection pool exhausts. Within minutes, every endpoint on your app returns 503 because they're all waiting on a dependency that will never respond in time. One slow service has taken down the entire application.

This is the failure mode that circuit breakers exist to prevent.

How latency spreads upstream

Most web servers handle requests with a fixed pool of threads or worker processes. When a dependency is slow, each in-flight request holds a thread for the full duration of the timeout. If your HTTP client timeout is 30 seconds and requests arrive faster than they complete, you run out of threads fast.

Once threads are exhausted, new requests queue. The queue fills. Eventually the load balancer or gateway starts seeing timeouts from your service, not just from the downstream dependency. To a user, your service looks broken, even the parts that have nothing to do with payments.

The cruel irony: a 100% failure rate would actually be better here. Fast failures free threads immediately. It's the slow failures that kill you.

The three states

A circuit breaker wraps calls to an external dependency and tracks their outcome.

Closed is normal operation. Calls go through. The breaker counts failures. If failures exceed a threshold within a rolling window, the breaker trips open.

Open means the breaker is tripped. Calls fail immediately without touching the dependency. This is the whole point: fast failure instead of slow failure. After a configured timeout, the breaker moves to half-open.

Half-open is the probe state. A limited number of calls are allowed through. If they succeed, the breaker resets to closed. If they fail, it trips open again and the timeout restarts.

That state machine is simple. The value is enormous.

Implementing it with opossum

opossum is the most mature circuit breaker library for Node.js. Here's a realistic setup wrapping an HTTP call to a payment API:

import CircuitBreaker from 'opossum';
import axios from 'axios';

async function chargeCard(customerId: string, amountPence: number): Promise<{ transactionId: string }> {
  const response = await axios.post(
    'https://payments.internal/charge',
    { customerId, amountPence },
    { timeout: 5000 }
  );
  return response.data;
}

const breaker = new CircuitBreaker(chargeCard, {
  timeout: 5000,                 // treat calls taking >5s as failures
  errorThresholdPercentage: 50,  // open when 50% of calls fail
  resetTimeout: 30000,           // wait 30s before attempting half-open
  volumeThreshold: 5,            // need at least 5 calls before stats matter
  rollingCountTimeout: 10000,    // rolling 10s window for failure rate
});

breaker.fallback((_customerId: string, _amountPence: number) => {
  return Promise.reject(
    new Error('Payment service is temporarily unavailable. Please try again in a moment.')
  );
});

breaker.on('open', () => console.warn('Payment circuit breaker opened, failing fast'));
breaker.on('halfOpen', () => console.info('Payment circuit breaker probing, attempting recovery'));
breaker.on('close', () => console.info('Payment circuit breaker closed, service recovered'));

export async function processPayment(
  customerId: string,
  amountPence: number
): Promise<{ transactionId: string }> {
  return breaker.fire(customerId, amountPence);
}

When breaker.fire() is called and the circuit is open, opossum calls the fallback immediately without touching your payment API. No thread held, no connection consumed.

Configuring the thresholds

The defaults are rarely right for your use case.

timeout should match the p99 latency of the dependency under normal load, with a small buffer. If payments normally complete in 800ms, a 2000ms timeout is reasonable. 30000ms is not: it just delays the cascade.

errorThresholdPercentage controls sensitivity. 50% means half your calls need to fail before the breaker opens. For a payment service I'd go lower, maybe 25%, because false positives (brief opens during recovery) are less damaging than letting failures pile up.

volumeThreshold prevents the breaker from opening on a single failure during low-traffic periods. Set it proportional to your expected requests per rolling window.

resetTimeout is how long you leave the dependency alone after a trip. Thirty seconds is a reasonable starting point. If your dependency has a known restart time, say a Lambda cold start or a container that takes 20 seconds to boot, calibrate to that.

Fallback design matters too. Don't silently swallow errors and return fake success: that creates data consistency problems. Return a clear error that your application layer can handle, whether that's queueing the request for retry, showing the user a specific message, or routing to a manual process.

Observability

A circuit breaker that opens silently is nearly useless. You need to know it happened and track how often it's happening.

opossum emits events you can push to your metrics system:

import { Gauge } from 'prom-client';

const circuitState = new Gauge({
  name: 'circuit_breaker_state',
  help: 'Circuit breaker state: 0=closed, 1=open, 2=half-open',
  labelNames: ['name'],
});

const circuitRequests = new Gauge({
  name: 'circuit_breaker_requests_total',
  help: 'Total requests through circuit breaker',
  labelNames: ['name', 'outcome'],
});

function instrumentBreaker(name: string, cb: CircuitBreaker): void {
  cb.on('close', () => circuitState.labels(name).set(0));
  cb.on('open', () => circuitState.labels(name).set(1));
  cb.on('halfOpen', () => circuitState.labels(name).set(2));
  cb.on('success', () => circuitRequests.labels(name, 'success').inc());
  cb.on('failure', () => circuitRequests.labels(name, 'failure').inc());
  cb.on('fallback', () => circuitRequests.labels(name, 'fallback').inc());
  cb.on('timeout', () => circuitRequests.labels(name, 'timeout').inc());
  cb.on('reject', () => circuitRequests.labels(name, 'rejected').inc());
}

instrumentBreaker('payment', breaker);

You also want circuit state in your health check endpoint. A health check that returns 200 while a circuit is open is misleading. Add a degraded state:

app.get('/health', (_req, res) => {
  const paymentOpen = breaker.opened;
  res.status(paymentOpen ? 207 : 200).json({
    status: paymentOpen ? 'degraded' : 'ok',
    dependencies: {
      payment: paymentOpen ? 'circuit-open' : 'ok',
    },
  });
});

Set an alert on circuit_breaker_state > 0 with a 2-minute sustained window. That gives you enough signal to investigate without waking someone up over a transient blip.

---

Most resilience failures I've seen in production weren't caused by single points of failure: they were caused by cascades. One dependency degrades, and within minutes everything downstream is on fire. The fix is almost always the same: fail fast, fall back gracefully, and keep the rest of the system moving. Seaworthy flags missing circuit breaker patterns automatically.