Context
A routine cost review showed one non-production account spending several times its usual amount for a few days, then stopping on its own. Nothing had shipped, nothing had alarmed, and from the outside the environment looked perfectly healthy.
That combination is what made it worth chasing. A bug that can burn money quietly in a rehearsal environment can do the same where customers are.
Investigation
- Cost data grouped by resource pointed at two background functions in one product. Neither was anything a user would ever see.
- Their metrics told two different stories. One showed a wall of invocations that started and stopped at exact moments. The other showed a steady hum of timeouts, around the clock, for weeks.
- The wall was a maintenance job that processed a batch and queued a message to continue with the rest, with no limit on how many times it could continue. The work it was retrying failed for the same reason every time, so the job kept scheduling itself until an unrelated change happened to clear it.
- The hum was a consumer with no dead-letter queue. One message that could never succeed came back after every timeout, thousands of times a day, indefinitely.
- A look at the database confirmed the scale was tens of thousands of items, not a handful.
Fix
- Cleared the stuck work at the source and purged the affected queues, so the loops had nothing left to chew on.
- Gave the maintenance job an iteration cap and back-off, so a batch that cannot make progress exits instead of rescheduling itself.
- Added a dead-letter queue to the consumer and switched on recursive-loop detection for the functions that fan out through queues, so a message that keeps failing is parked rather than replayed.
- Added alarms on queue age and depth, on invocation counts against a rolling baseline, and on cost anomalies for the account, so the next storm is a page in its first hour instead of a line item at month end.
Outcome
The loop tried again a few weeks later, before the cap had reached every environment. This time the alarms caught it the same day and the spend stayed small. That second run also settled any debate about whether the fix belonged everywhere.
The permanent hum stopped the day the dead-letter queue landed.
Lessons
- Any job that schedules its own continuation needs a cap, and the cap needs a metric. 'It will finish when the work runs out' assumes the work can run out.
- A queue without a dead-letter queue is a retry loop waiting for its first poison message. Add it before the first consumer, not after the first incident.
- Cost is a lagging signal. Queue age and invocation baselines are leading ones. Alarm on those.
- Non-production spend deserves the same anomaly detection as production. It is where production bugs rehearse.