The loop that cost $1000+ overnight - and why 'improve test coverage' was the actual bug

Hot take: unattended agent loops are great, right up until you’re the one paying for tokens. We’ve seen loops costing $1000+ in a single run, which is a real problem if you’re operating under any kind of budget.

What actually happened

A team we spoke to ran a simple “improve test coverage” loop. The goal was too broad and had no stop condition:

“Skip if a <redacted> agent is in progress. Improve test coverage on <redacted>. PR the results and auto-merge if you deem it low risk.”

They kicked off the cron job in the background during the work day. Nothing much happened for most of the day, because other agents were already active on the same repo and it kept skipping. They forgot about it.

Then, overnight, with nothing else running locally on that repo, it got busy - and produced 30+ PRs by morning.

Why it ran away

Because the goal was “improve,” it could always find more to do. Test coverage doesn’t have a natural stopping point short of 100%, and no real production repo is anywhere near 100%. There was also no guardrail or stop condition telling it “you’ve done enough, go rest.” Just a poor agent with an open-ended mandate and nothing to bound it.

Equally important: because it spun up a fresh agent for each run, it paid the full cost of context-hunting every single time - each agent had to re-assess a meaningful chunk of the repo from scratch to work out what was going on before it could even start. That’s a lot of tokens spent on repeated discovery work that a persistent orchestrator wouldn’t have had to redo.

The team told us most of those 30+ PRs were at least mildly valuable individually. But given the choice up front, they wouldn’t have knowingly spent $70+ in tokens - the average cost per PR - on most of them.

The actual lesson

This isn’t really an argument against unattended loops. It’s an argument that this loop wasn’t designed, it was an experiment that got away from its owner. A properly designed loop needs, at minimum:

• A bounded goal. “Improve X” has no stopping point. “Bring X’s coverage from 61% to 75%, then stop” does.
• A stop condition that isn’t just the goal being satisfied. Something explicit that says “enough for this run,” independent of whether more improvement is theoretically possible.
• Shared context across runs, rather than a fresh agent re-discovering the repo from zero every single invocation. That’s pure overhead being paid repeatedly for no benefit.
• An orchestrator with prioritisation, so it’s not just grinding through whatever it finds first, but triaging opportunities against some sense of what’s actually worth the tokens.

It comes back to the classic effectiveness-versus-efficiency question, and this loop is a clean illustration of both sides. Effectiveness: very high. It did exactly what it was asked and improved test coverage by roughly 8%. Efficiency: nearly zero. It spent something like 10x what the team would have chosen to spend if anyone had been asked.

We’re all going to have to get comfortable re-asking that old question for every agent loop we set up - unless you’re operating with an effectively unlimited token budget, in which case, congratulations, and also please design your loops properly anyway.