Circuit Breakers Alone Won't Save You

Our service was choking. We thought circuit breakers would save us.

Turns out, that’s only half the story. 😅

We called a third-party Service B from our main flow. Service B was unreliable — down often, and when it wasn’t down, responses were taking 30+ seconds.

The pattern people know

A circuit breaker watches for failures. After N consecutive failures, it opens and returns errors immediately without attempting the call. After a cooldown, it lets a small batch through to test recovery.

Clean. Elegant. Widely taught.

Here’s what’s missing: the circuit breaker can only open after failures accumulate. If calls are timing out slowly — hanging for 30 seconds each — you accumulate 60 seconds of blocked threads before the circuit trips. By then, your connection pool is exhausted.

We added a strict 500ms timeout first — if Service B didn’t respond in time, we cancelled the request. Failures dropped, but not enough.

Circuit breaker states: Closed → Open → Half-Open, with 25% traffic ramp-up

The three states

CLOSED — everything normal, requests flow through to Service B.

OPEN — too many failures detected, we stop all requests to Service B and return errors immediately.

HALF-OPEN — after a cooldown, we test with a small batch. Instead of going all-in, we gradually ramped traffic:

25% → 50% → 75% → 100%

If failures increase → back to OPEN. If Service B holds stable → fully CLOSED.

The lesson the hard way

A slow dependency is more dangerous than a fast-failing one. Fast failures trip your circuit breaker. Slow responses drain your resources while everything looks green on your metrics.

The resilience stack

After this incident, resilience became three layers working together — not alternatives:

Layer 1: Timeouts — every external call gets an aggressive timeout. Not the framework default. A number you calculated. If it hasn’t returned in 500ms, kill it. Fast failure > slow success.

Layer 2: Circuit breakers — now that you have timeouts, failures accumulate quickly. Configure breakers to count timeouts too, not just exceptions.

Layer 3: Traffic ramp-up — when a circuit closes, don’t immediately send full traffic. A service that just recovered is fragile. A sudden burst often kills it again.

Request → [Timeout: 500ms] → [Circuit Breaker] → Service B
                                    ↓
                         [Ramp-up on recovery: 25→50→75→100%]

Smart traffic ramp-up: 25% testing → 50% gradual → 75% confidence → 100% fully closed, failure logs down 40%

Results

After wiring all three together:

Failure logs reduced by 40% ✅
Service stayed healthy under load ✅
Next bad-dependency day was boring — pricing calls failed fast, circuit tripped, fallback showed, nothing else was affected

The pattern that works

Timeouts make failures fast. Circuit breakers prevent calling a broken service. Traffic ramp-up protects recovering services. All three, in sequence.