80% of our APIs hit Redis, directly or indirectly.
Slowly, Redis is becoming a single point of failure.
During a tea break with my manager, I went to the vending machine for biscuits. It had a technical glitch and stopped working. So I walked to the cafeteria, and for the first time, I saw it crowded.
The vending machine failed, so everyone fell back to the cafeteria.
I told this to my manager. He quietly said:
That’s your DB when Redis goes down for 30 seconds.
That stuck with me. What if Redis goes down?
Thousands of requests, complex queries, calculations, reads, and writes, all landing on the database at once. That’s the thundering herd problem. The database instance slows down and eventually dies.
What can prevent this?
1. Short Redis timeouts
Fail in milliseconds. Don’t block every request thread waiting on a dead cache.
2. Local in-memory fallback (Caffeine)
Keep hot keys warm locally. The trade-off: this is wiped whenever pods restart.
3. Circuit breaker + load shedding
If Redis is down, don’t forward everything to the DB. Serve stale data and drop non-critical calls.
The cache made us fast. It also made the DB forget the real traffic.
If Redis is down, your DB shouldn’t be.