What building this site taught me about AI agents

I started this site partly as an experiment. I wanted to know how much of a real, public website an AI coding agent could build if I gave it the work in batches and reviewed each batch before allowing the next. I meant a working site, with a database behind it, a form that sends email, published content, rules about what the site may record about a visitor, and a deployment pipeline. The question I began with was about capability. How much can the agent do?

A few batches in, the question had changed. The agent could do a great deal. What I could not do, at first, was tell whether the thing it had built was the thing I had asked for.

What the agent could do

The volume of competent work was the first surprise. Given a written brief, the agent would produce a page, its styles, its tests and its documentation in one pass, and most of it would be right. It kept to the brand tokens. It wrote tests that read the source. When a batch called for forty or fifty single-change mutants against the test suite, it ran them, listed the survivors and tightened the assertions. I had expected to spend my time correcting code. I spent very little of it that way.

Capable, and confidently wrong about the world

What I had not expected was how often the agent was correct about the code and confidently wrong about the world the code lived in.

The clearest case was the review server. The site builds its content in from a database, so what a page shows depends on which content source the build used. The development server, with no source set, reads seed records, and for a while it sat on the same port the production-parity build was supposed to own. Twice, a review compared localhost with production and concluded that pages were older or newer than they were, because it was looking at sample content on the wrong process. The code was fine. The environment was misleading, and nothing in the agent's checks asked whether the server on that port was the one it thought it was.

The same pattern showed up elsewhere. A scroll animation could work, in the sense that every test passed, while the editorial scene behaved wrongly on a real laptop because the stage was not the height the tests assumed. An artwork could be integrated correctly and still look wrong, because a near-square frame contained a wide drawing and left white bands nobody had asked for. Mutation testing found assertions that could not fail: a pattern meant to catch stand-in image services that also matched an ordinary word in the source, and another that matched any media query rather than the one that mattered. None of this was visible from inside the code, and the tests were the wrong instrument for finding it.

The common cause was what the agent could observe. It had the sources and the tests. It did not have the port table, the rendered page at 1366 pixels wide, or the memory that a previous session had left a stale process running.

The environment became part of the product

So the work shifted. Instead of asking for more autonomy, I spent the batches building the surroundings the agent worked in, and those surroundings turned out to matter more than the prompts.

Some of it was rules written down where the agent would read them first: which Firebase project is the only permitted target, which files are generated and must never be edited by hand, which pages must stay text-identical to production. Some of it was mechanism. Production-parity review became one command that pins the content source, builds the current tree, serves it on one fixed port and prints the commit and build id it is serving, so a review that starts on the wrong server fails at the first line rather than at the end. Seed development moved to a different port for good. Tests were extended to read the data modules directly, and mutation runs became a normal part of a batch rather than a special event.

Git carried a lot of the weight. Every batch became one local commit on a chain above the last deployed commit, with the starting SHA and the remote SHA recorded before any edit, so the agent could check that nothing had moved and I could see exactly what a batch contained. Nothing was pushed until I had looked at the rendered result. A rollout was never assumed to have worked because the build succeeded; the live site was fetched and compared afterwards.

Each batch also ended with a stop. The brief said where the work finished and what to report, and the agent stopped there. The stop is what makes a session something I can review instead of something I have to reconstruct afterwards.

Where the review went

I had assumed that the review burden would fall as the tests grew. It fell in one place and rose in another. I stopped reading implementation detail line by line, because the tests and the mutation runs were better at that than I was. I spent more time on intent and on behaviour: is this the page I described, does it read well at 320 pixels, does the featured panel look balanced, is the article's argument the argument I hold. Those are questions the agent can help answer, with screenshots and measurements, but they are not questions it can settle. Somebody has to accept the work, and that person has to be able to say why.

The other thing that rose was the cost of a wrong assumption early. A misread brief costs a batch. An unexamined environment costs a batch and the confidence of the next one. Most of my effort went into making those assumptions checkable before the agent started, which is a strange place to find yourself when you set out to test how much code a machine can write.

What an agent needs

After enough batches of this, my list of what an agent needs in order to do dependable work has very little to do with the model:

• a bounded task, with an explicit place where it ends;
• reliable context: the rules, the current state, and the history that explains both;
• constraints it cannot argue its way around, such as a single permitted deployment target;
• observable outputs, meaning a rendered page, a measured layout or a served build id, beyond a passing test;
• tests that can fail, proven by trying to make them fail;
• a source of truth for every fact the work depends on;
• a stop condition;
• someone accountable for accepting the result.

Every item on that list is something I supplied. The agent can be reminded of them and follows them well, but it cannot originate them, because they are decisions about what I am prepared to be responsible for.

What the experiment turned into

The experiment answered the question I started with, and the answer was more than I expected. Then it replaced that question with a better one. How autonomous the agent is turned out to matter much less than whether the system around it, the rules, the tests, the checkpoints and the person at the end, lets me trust what it produces. On this site, that trust was built one batch at a time, and most of the building happened outside the code.