Show me what you think we built

In each of the presentations I’ve made on this method of working, there is one question which comes up more frequently than any other. When I display a document that has been generated, typically a high-level solution design, someone will ask, “Do you actually read all of that?”

Yes, in a deliberate and detailed manner, step by step. This is the most important task I have now, and this article explains why that is.

This wasn’t meant to be the next article in the series, but on Friday Anthropic’s Applied AI team published The AI-native SDLC playbook. This playbook is well worth reading: it independently puts forward a very similar methodology to this series of articles, which is the best kind of validation. A few parts of my process went beyond what Anthropic published, and this article describes what I think is the most important unique part of my process…and that made the subject topical enough to jump the queue. I’ll get back to regular programming next week.

It started with a customer meeting

In April 2026, TrolleyRelay was a proof of concept and a backlog. A validation meeting with the first customer was due, and I had two requirements: the questions only the customer could answer, and an explanation of what we had built so far, what we had proven to work, and our plans for next steps.

The initial list was already there since my product-owner agent, the same agent mentioned in the previous article, had been maintaining it by recording all the open questions that required a response as they built up in the backlog; therefore, I asked Claude to convert the entire thing into an official document aimed at customers.

I was surprised by the output. It was well structured and almost ready to use, so the day before the meeting I planned to review it and improve it. However, the editing process took me in a different direction. I kept discovering flaws, and these weren’t simply spelling mistakes; they were errors based on assumptions: some claims rested on things I didn’t believe, and one or two rested on things I did believe even though the evidence didn’t support them.

This practice originated with that document, since it wasn’t merely a deliverable but the first instance in which I could see, in the form of text and diagrams, what the AI believed we were building, and the differences between its model and mine became easy to identify.

The questionnaire that I took to the meeting took that idea a stage further by having, in italics underneath each question, any assumption which we had already formed ourselves. As an example, for the question on catalogue size, “dozens, hundreds, or thousands” was given as the expected format for the answer; the customer’s answer was that the number was between five and ten thousand, and that products are archived rather than deleted so that nothing loses its history. That one amendment altered our synchronisation architecture before we built any part of it.

A document of that kind is a three-way diff; it includes my model of the system, the AI’s model, and the customer’s reality, with all the differences being visible on the same page.

You cannot test architecture

Behaviour has tests. My pipeline enforces them ruthlessly: every change ships with tests, coverage can’t drop, and the checks prove data mutates the way I expect. None of that proves the system is shaped the way we both (me and my team of agents) think it is. There is no unit test for “the architecture in my head matches the architecture in the agent’s head”. Peter Naur made a version of this argument about human teams in Programming as Theory Building (1985; a scanned copy circulates freely), holding that the real program is the theory its builders carry in their heads, and that it is the theory, not the code, that decays and gets lost. In this process, half of that theory is held by an agent.

In an AI-driven process, the mismatch is dangerous, because a false assumption doesn’t sit still. It carries into the next session, then the one after that, and quietly compounds until something visible breaks far from the original misunderstanding. This is documented model behaviour rather than superstition. Zhang and colleagues examined it in How Language Model Hallucinations Can Snowball (2023), finding that language models “over-commit to early mistakes” and then produce further errors to justify them. This can compound in an AI session but the issue gets so much worse in an AI-SDLC process which depends on an ever-increasing corpus of LLM-generated documentation, persisted across multiple sessions over days, weeks and months.

Interrogating the agent question by question does not scale; you only check the assumptions you already suspect. Reading a document it generated about the whole system checks assumptions you didn’t know you were carrying. Divergence surfaces as a comment on a page instead of a surprise in production.

So when you read the generated document, it is the architecture test, and it addresses the question I always get. The reading is slow, since that is the point. It is the test being carried out, and it represents the most important human review.

A document needs something to stand on

A necessary precondition for any of this to work must be mentioned here even though later articles deal with it in much more detail.

You can’t come up with a reliable solution design out of nothing; it must be developed from the underlying documentation maintained throughout the development process, such as the project rulebook, decision records, the backlog, and the domain glossary. Maintaining that body of documentation and constantly checking the lower-level documentation and the code against reality is an ongoing responsibility in this process. The high-level document forms the top level of a pyramid, and this article focuses on that top level; the various layers below will each have their own articles later in the series.

If your documentation is lacking or out of date, this approach probably isn’t the right entry point for you. You need to build the foundations first, which will be part of a future article.

A document designed to be read

The complete solution design didn’t arrive until a month and a half after the first customer document, following the pattern tested on some smaller customer-facing docs. It is a single, self-contained webpage structured for two different audiences: the front section presents the benefits to the merchant, while the back section provides the full technical architecture. The document is prepared so that it can be shared with a future customer under the terms of an NDA. The requirements in the README state it is “written to not diverge from what is built or planned”.

To ensure that, I established three principles.

• First, each capability has a status label, and the document is prepared for a specific targeted release (version one must include this, that, and the other thing). Nothing refers to the future in a comfortable future tense; a capability is built, partly built, or planned, and the document states which one.
• Second, the claims are cited. In the most recent iteration, sixty-three citations link capabilities to the decision records and backlog items they rest on, and the backlog citations carry the live status of the item they point to. A capability that claims to be built while citing an item whose status says otherwise is visibly wrong at a glance: the document audits the tickets, and the tickets audit the document. A toggle hides every citation, producing the lighter customer cut from the same file.
• Third, it is deliberately more polished than it needs to be. Black-on-white Markdown is fine for agents, and fine for me in small doses. This document is styled, coloured, and includes diagrams; partly because a customer may see it, but largely because it is for me: the polish makes the slow, detailed read something I actually do rather than something I intend to do. There is a reason for the colour; it’s more human-friendly.

What was the actual impact?

The first full review was published at the beginning of July. I treated the document as I would a solution design handed to me by an architect and examined it against all the changes made in the six weeks since it was written. As a result, I produced eleven pull requests. The section on billing was changed from planned to built because our backlog hadn’t been properly updated. We also identified a claim in the roadmap about two-way inventory sync (a feature that had since been removed). However, one finding had a real impact on the product, not just the document, and that was the most important outcome: when we reviewed the onboarding process as a series of diagrams, it became clear that our onboarding process was defective. Instead of asking merchants to approve payment, we should first test that their ShopFront plan supports the integration; it would have been a terrible user experience to pay first, then find out your plan wouldn’t support our system without additional ongoing costs. I think in terms of diagrams and graphs, which is why my agents can now show their ideas in a form that is easier for my brain to understand.

I version my solution designs against my Architectural Decision Register (more on that in a future article) and review the design periodically. These reviews also leave an artifact that records what the revision found but did not resolve: work the document describes that is not yet built, and decisions the review deliberately left to a human. It works as a ledger, not a graveyard: a formal decision record closed one of its open questions three days after it was filed. The review does not just fix the document and the underlying platform. It writes down what it refused to fix, and who owns each leftover.

The last ten per cent

There is another payoff, and it might be of greater importance to more people than the “architecture test”.

The most common failure mode I see in AI-assisted projects is not broken code. It is projects that do not finish. Getting to 90% is easier than ever; the last 10% is still where projects quietly die, as it has always been.

After the second review, the review-record file became a statement of what still remained between the current system and the release described in the document, checked by a human to catch misunderstandings or incorrect assumptions. This list then became the target for closing out the last 10% and getting the system live.

For a project like this (that is built step by step, in an agile manner, as an improved understanding emerges), to get this type of visibility, I would have to live in JIRA, but that would only capture part of what I have achieved here and would have missed half of the issues this process found.

The practice that travelled the other way

In this series, almost everything followed one direction: they were tried out in the project first, and then the winners were recommended to the enterprise; the opposite was true here.

Solution design documents are a standard part of architectural practice, and I have written them by hand for many years, both for internal review boards and in the context of enterprise sales, so the customer’s security and architecture teams can examine them before implementation. The review process, which involves a person in a position of authority carefully reading the design and questioning its assumptions, is dull yet effective.

This project has changed both who prepares the document and how often it is done. This practice was introduced into the lab from that environment, with the lab’s own contribution being that an agent looks after the document periodically and that the wiring enables it to self-audit. I haven’t introduced this extended solution-design idea into the enterprise world yet, but I think it will bring benefits there too.

Alongside the playbook

Anthropic’s approach is also based on the use of artifacts: you have to capture the intent, compress that intent into a specification, expand the specification into a plan, and then keep the artifacts under version control. So if you are introducing an AI-driven process, it is indeed a good one.

The difference lies in the direction taken. The artifacts in the playbook point forward: each one passes control on to the next stage and then considers its task to be completed. All the content in this article points backward: it is a renewed assertion of what currently exists, connected to the live state, and is read by a human in order to identify the places where it is incorrect. Both of these approaches have their uses. The forward chain carries out the intent; the backward document checks for understanding. If you already have a similar AI-SDLC playbook in operation, I recommend that the system describe itself regularly in a document you actually read, and that you treat every discrepancy you discover as a defect detected early.

Do you actually read it?

So, to the person in the audience who asks whether I actually read what the AI generates: yes. All of it, slowly, against everything I believe about the system. It is the last job I would automate, because it is the one where the machine checks me as much as I check the machine.