Half Our Events Just Vanished

Half our production events were lost — because of one small mistake. 😅

I used Codex to consume RabbitMQ events from another service. Gave it full context + both repos. It generated clean, working code. Local testing? ✅ Perfect.

Then production happened. Some events were processed. Some just… vanished. No errors, no crashes. Just missing data.

What went wrong

Both services were consuming from the same queue.

In RabbitMQ, a single queue distributes messages between consumers — it doesn’t duplicate them. So instead of both services getting all events, they were unknowingly splitting the load.

Analytics was seeing ~50% of events. Real-time dashboards were seeing the other 50%. Neither failed. Both appeared healthy. Silent data loss.

The correct setup

Each service should have its own queue, bound to the same exchange with the same routing key:

Exchange: vehicle_events (topic)

Queue: analytics_events   ← Service A consumes
Queue: realtime_events    ← Service B consumes

Same routing key → exchange fans out to both queues
✅ Every service gets every event

Event hits exchange → routed to multiple queues → each service consumes independently.

What I’d do differently

Before accepting AI-suggested architecture, write out the data flow requirements explicitly:

• Who needs to receive each message?
• Does each consumer need an independent copy, or are they competing instances for scale?
• What happens if one consumer is slow — does it affect others?

These questions reveal whether you need competing consumers (scale) or independent queues (fan-out). AI can design either pattern — but you have to tell it which one you need.