Build the quality floor before the autonomy

On 11 April 2026, four days into building TrolleyRelay, an agent-managed backfill of products from ShopFront into a newly cleared Shopify store reported that all the products had been successfully created, even though nineteen of them still did not actually exist. After spending a while troubleshooting, I located the cause of the silent failure by writing a diagnostic that compared our local table with the actual contents of the store, since nothing along the normal flow of operations revealed the issue. The API returned a success message, the product writes were “successful” and the logs showed no errors. The reason was that, for about forty seconds after a bulk product clear, the store would return valid-looking product IDs for products that it then quietly discarded, and all nineteen phantom products were the first ones to be processed after the clear.

The failure mode of a fast implementer is not code that fails loudly. It is code that succeeds falsely. It was time to make sure any failures were loud.

A proof of concept, right up until it wasn’t

This project did not start as a product. Like many projects, it began as a proof of concept to see if what I wanted to build with Shopify/ShopFront was possible. For the first four days, there were no tests, and that was the right call. Prototype mode is legitimate: prompt, accept, iterate, discard. Vibecoding has its time and place, and this was it, because nothing downstream depended on the output yet.

The phantom products ended that phase. The pivot, in my words at the time, was: “OK, it works. Now it’s time to start building properly.” Same agent, same codebase, same week. What changed was not the code’s quality but that the output was about to be owned: production behaviour, a real store’s catalogue, and eventually other people’s inventory and money. The quality floor this article describes is the line between those two modes, and the rest of the series builds on this.

The day the project found a defect it could not see, it built the apparatus to shine a light on these issues. That is the meta-process described in the opening article running for the first time, four days in: the incident arrives, and the rule it produces is dated, specific, and traceable to the failure that earned it.

While I had been manually testing and reviewing every pull request until this point, this was when I started to step back and give some autonomy to the fleet of agents. Now they could manage their own tests, with my CLAUDE.md adding a testing section and the line: “Every code change must ship with tests in the same commit/PR. Do not write code and add tests later.”

The bottleneck does not go away. It moves.

Why is the floor the first investment before loosening oversight? Because an agent can produce more code in a week than a human can meaningfully review, and something has to absorb the difference.

Anthropic’s AI-native SDLC playbook, published this August, states plainly: “The bottleneck moves to the steps to the left and right of the build phase” - mainly plan, review, and test, which still run at human speed. On review specifically: “Reviewing each line by hand made sense when a person had written it, but it can’t keep up once agents write most of the diff.” I read that with satisfaction because it is the wall this project hit in its first week, documented above. I had already converged on the same conclusions before the playbook was released, which is good evidence for the process I’m outlining in this series.

Build shrinks; everything around it still runs at human speed. Diagram from Anthropic’s AI-native SDLC playbook, reproduced with attribution.

None of this is new physics. Goldratt’s Theory of Constraints says when you elevate a constraint, the constraint does not vanish, it migrates. Little’s Law and Reinertsen’s product development flow say queues are where the cost lives. My observation from running this rollout and the enterprise one alongside it is that work in progress is still the problem it always was; it just piles up in new places behind verification that still runs at human speed.

The floor was the first response to that relocation (spoilers, we’ll discuss this more in future articles). If build is no longer the constraint, we need to figure out how to feed and validate it with as much automation as possible, so scarce human hours focus on what only a human should assess.

If the agent writes its own tests, how do you know it is testing the right thing?

This is the objection I hear most, and the honest answer is that the right thing matters less than unexpected change. Since April I’ve said this phase was about “quality” but that is subtly wrong; here I’m not enforcing quality, I’m allowing an agent to detect unexpected change. A more honest name would be the “regression floor.”

Unit tests are not about getting it exactly right. They are for managing change. They make up the majority of what agents write, and their job is to ensure that when an agent changes something, only the expected thing changed. Unexpected change hurts at agentic levels of throughput, and the suite exists to catch it. A ratcheting coverage floor buys exactly that, and it is the only thing I really care about catching. It’s why I wouldn’t consider building with an untyped language with agents: being able to catch regressions/bugs early, before they happen, is what matters.

Correctness and functionality are established differently, with heavy human involvement. My focus is on the layers above: manual validation, full integration tests (written by an agent but validated by me), and end-to-end tests…all establishing the system delivers exactly what it should. The split is deliberate: agents own volume at the unit layer, I own judgment where correctness lives. This is the bottleneck made personal: my hours did not disappear when agents took the build step; they moved to steps that still run at human speed, exactly where Anthropic’s playbook says the constraint now sits.

There is one exception: in sensitive areas (financial and billing classes are classic examples) I care about the unit tests themselves, not just the change signal. This does not trip up the process because touching those areas automatically triggers a mandatory human review (more detail in a future article). In enterprise settings, we usually have coverage targets and require human review for all merges. I think we will move to a world where we target what requires human review and otherwise auto-merge once an extensive CI suite passes.

The rest of the split transfers to that setting too, because “I manually validated it” does not scale beyond organisations where the operator is also the owner. Name which assurance comes from the suite (regression detection), which from human review (correctness in areas that can hurt you), and which from validation (does it do the job). Teams that fail to make that split either trust coverage to mean correctness or reject agent-written tests wholesale.

The next question that’s usually put to me is whether I trust the agents. The answer is no, I don’t. In fact, I’ve never 100% trusted any developer who has ever worked for or with me, including myself…which is precisely the reason why we have peer review, automated testing and the entire SDLC process to begin with. The content of this article is not new bureaucracy created because the agents are uniquely untrustworthy. It is the same kind of institutional management of human fallibility that every serious engineering organisation already employs, but now it must be carried out more strictly, checked by machines rather than by humans, and we must ensure it can be audited afterwards.

If you believe in peer review, you already believe authors should not be trusted to certify their own work, and an agent author changes the volume, not the principle.

People also write their own unit tests. It is not disputed that a developer’s own set of tests does not prove their code to be correct; rather, it is relied upon to detect any unintended changes. What is new about agents is the scale, and it is exactly for this reason that the mechanism has to be machine-checked rather than involving a human operator.

Before proceeding, let me mention one point regarding the unit suite: it is merely the first tenant of that floor. The same CI gate subsequently developed other types of teeth - structural tests that make the codebase conventions unbreakable, drift linters that check what a config states against what is actually deployed, and a build step that demonstrates the code does not merely typecheck but actually bundles. These various measures carry out functions that are different from those of regression detection and will each be covered in separate articles later in the series.

What this floor bought, and what it did not

What it bought is the chart below: elevenfold growth in the code base, a test suite larger than the code it guards, and an agent trusted to write most of the code, because the “quality” holds whether or not any human reads every line. The test suite overtook the production code on 24 May and never gave the lead back: 78,556 lines of tests guarding 75,368 lines of code at the time of writing. No human read every line. The floor made that volume affordable to supervise.

What it did not buy: in June, a manual audit (i.e. reviewing the solution design from an earlier article) found an issue with a safety feature to avoid events echoing between the two systems, causing an infinite loop. All the code had been written but it hadn’t been hooked up to the key functions that needed it. Full coverage of that code and a green pipeline were both true while it sat there, unused. Coverage proves a line executed under test. It cannot prove the line is reachable from anywhere that matters in production. Floors are necessary and not sufficient, and a later article in this series is entirely about that class of false green.

The floor comes first

The takeaway is that you need to have a method for managing unexpected change before you can expand autonomy…not because autonomy is dangerous, but because a coverage/regression floor is the only thing that makes autonomy affordable to supervise. Every step back I took to allow more autonomy (which is what the rest of this series covers: grill gates before implementation, reviews of code no human wrote, judgment gates) is spending trust that this phase banked in advance.

And if you want it in one sentence: the quality/regression floor is what “I don’t trust anyone, including myself” looks like when you pour it into automation.