Catching the agent: mistakes from a real refactor
Nine days, about 110 commits, an AI agent doing most of the typing. These are the mistakes it made that reached a commit or a screen, how each one was caught, and the guardrails that came out of them.
- When it happened
- Published
- Measured
- , read from the commits and the session transcripts
- Stack
- Claude Code, PHP 8.4 (CodeIgniter 4.7), ClickHouse, MySQL, headless Chromium
- Also in
- Português
The work
The service generates proof-of-play documents: for an ad order, a PDF and spreadsheets saying what played, where and when, read from ClickHouse. I rewrote it with the agent between 27 August and 4 September. The new version ran in parallel on the same host, under /v2/, with the old one as reference, and became the official one in the first days of September. My rule during that time was simple: push yes, pull request no, nothing goes live until it's finished.
Most of the code was typed by the agent, and I directed, reviewed and ran things. Below are its mistakes that reached a commit or a screen, in its own words when it wrote them in a commit message.
The 15,000-row cap
Big PDFs were dying. One order with 32,958 rows ran for 12 minutes 52 seconds and ended in Allowed memory size exhausted, with HTTP 200. On 28 August the agent fixed it by refusing anything above 15,000 rows with a 413 and a message. The reasoning looked good for me: the PDF library used 131 to 137 KB per row, so with 2.5 GB the real limit was around 19,000 rows, and 15,000 stayed below.
On 31 August I opened a real order with 17,394 rows on the parallel deploy and it was refused. That should not happen. The old system already generated big ones in the background, with the page polling for progress, and I knew by using it that it worked. The commit after that says it plainly: "My cap of 15,000 refused before it started." With the client cut at 120 seconds by the CDN in front, the background generation kept going until the PDF was ready, 1.88 MB.
What mattered was the lock, and the row count was only a proxy for it. The generation holds a lock with a TTL of 900 seconds, and after that a second request starts the same PDF in parallel, two processes of 2.5 GB on the same file. From the measured point, 17,394 rows in 570 seconds and a curve close to n^1.75, the cap moved to 20,000 rows, which leaves 170 seconds of margin for a loaded machine. The agent optimized for what it could see in the code. I knew how the old system behaved in production, and every test passed with the wrong cap.
A try/catch that catches nothing
That cap came after another mistake. An earlier commit said a try/catch solved the orphan lock when generation died. It didn't, and the agent's next commit starts with "It was wrong, and the error was mine". Running out of memory or hitting max_execution_time is a fatal error in PHP, and fatal errors are not Throwable:
<?php
// php -d memory_limit=32M oom.php
try {
$a = [];
while (true) { $a[] = str_repeat('x', 1024 * 1024); }
} catch (\Throwable $e) {
echo "caught\n"; // never printed
} finally {
echo "finally\n"; // never printed either
}
Even a shutdown function didn't help, because the framework's exception handler calls exit() first. We forced an OOM with 64 MB to confirm it. After a fatal error there is nothing left to clean up, so the only way out is to not get there: count the rows before, and send the big ones to the background path.
A spinner that downloaded HTML as PDF
I asked for a spinner while documents generate, and the agent changed a plain into fetch plus a blob to turn the spinner on and off. But the same route returns a PDF when the document fits and an HTML page with options when it doesn't, and now any answer was saved as .pdf. The old link worked by accident: the browser just showed the page. This one reached production for a day. Commit: "Regression of mine". The fix now checks the Content-Type and, when it's not a PDF, lets the browser navigate like the link did before:
const res = await fetch(url);
const type = res.headers.get("Content-Type") || "";
if (!type.startsWith("application/pdf")) {
res.body?.cancel();
window.location.href = url; // do what the link did
return;
}
saveAs(await res.blob(), filename);Numbers that looked better than they were
"Is this real, a PDF in 5.6 s?" No. The 5.6 s was the time of the refusal, the 413. Real generation for 2,366 rows was 19 seconds.
A subagent reported one function going from 8.2 s to 0.4 s. It had measured it with 3,640 locations, and the real call passes 18 and costs 6 ms, so that gain was thrown away. Another measurement was contaminated by a second review clicking around the same app against the same ClickHouse, and the rule that came out is that counting queries in a shared log needs isolation first.
My favorite is from another project in the same month, a replica of a legacy export. The validator's headline said 97.76% of fields OK. Right below, the report carries the honest number, 95.82%, because the headline only counted fields that matched; buckets that never paired were out of the denominator. With the wrong grain it can show 100% without anybody comparing a single row. And 263 fields stay as NO_PROOF: the only available explanation for them can't be falsified, so they don't count as explained.
Smaller ones, same pattern
- "It's live", with the server one commit behind. I compared screenshots before and after, and the agent went to look at what was really rendering.
-
rm -fon a variable, six or seven times, to clear a cache. I told it I've seen one of those run on a server with an empty variable, starting from/. Its answer: "I was the one who chose the destructive tool for a problem that wasn't destructive." -
An inverted date window gave an empty proof of play. ClickHouse accepts
x <= day AND day <= ywithx > yand just returns nothing. In the same review, a daily chart was 24 hours late because an ISO date was parsed as UTC, and in UTC−3 the 19th showed as "18 Aug". Seven defects came out of that one review. - A PDF cache key that could collide between two customers, so one could get the other's document. Found by the adversarial review, before production.
- Scope: the agent wanted to add a second content partner to the project. Nobody asked for that, and when I questioned it, it wrote in the notes that it was an inference of its own.
What caught them
A baseline from the old system, captured before touching anything: 13 to 16 HTTP responses compared phase by phase, with digits normalized because new data arrives every 10 minutes. Old and new side by side on the same data. Real orders, the big ones especially, because tests only cover what you thought of. A reviewer subagent whose task is to refute and not to agree. And the question "is this real?" whenever a number looks good, which costs nothing.
Each mistake I catch becomes a written rule with the why, and the agent reads it on the next session, for example "never say it's deployed without looking at the screen". The agent changed the speed of this refactor. Without it, this cleanup would take me much longer, maybe long enough to stay in the backlog forever. Without the baseline and the real orders, the 15,000 cap goes live and some customer finds it first.
Was this useful?
Thanks. It helps me decide what to write next.
I design production systems from zero and I take over the ones already running. Either way they have to hold up when the business grows. I co-founded a digital out-of-home company and built its technology for ten years; my partners raised the money and ran sales.
Your business is growing faster than your systems? Let's talk →