The agent fixed the total it could see

The task was small and unambiguous: apply a ten per cent discount to the order total everywhere it is computed, consistently.

The total was computed in two places. billing.py summed order["lines"] as quantity times price. analytics.py summed order["items"] as count times cents. No function called both, no module read both keys, and neither file mentioned the other's vocabulary. That is an ordinary state for a system a few years old, where two teams needed the same number for different reasons and each wrote it in their own terms. Both functions returned 1300 for the test order, so a correct change left both returning 1170.

I gave the change to two models from different vendors, Claude Haiku 4.5 and GPT-5.4, thirty times each. The agent could see the billing module and two others. The analytics module wasn't in its context.

Sixty trials, sixty half-changes

Every trial patched invoice_total in billing, returned it as the whole change, and left revenue_for_order computing the undiscounted total. None of the sixty went looking for a second site.

Each of those changes would have passed review. The code was correct where it was written, the tests in the touched module passed, and nothing in the diff hinted that a second total existed. The only thing that could have caught it was a person who already knew about the analytics site. That is coherence debt being created in a single commit, by an agent behaving reasonably.

What actually fixed it

The first version of this experiment compared two arms: the blind agent, and an agent given a derived structural model of the system that named both places the total was computed, along with the analytics source the model pointed to. The second arm got it right thirty times out of thirty on both models, and I read that as evidence that the derived model was doing the work.

That reading bundled two things together, the map and the code it pointed at. So I ran the four combinations separately, thirty trials per cell on each model.

Access was the whole effect. With the analytics source in reach and no structural model at all, both models made the change correctly every time. Adding the model on top added nothing this fixture could detect. My first reading was wrong, and the write-up in the repository now says so.

The more useful result is the cell where the agent was told about the second site and couldn't read it. GPT-5.4 patched billing and stopped, every time. Claude Haiku 4.5 did something worse: it rebuilt the analytics module it had never seen from the two lines of description, got the discount arithmetic right, and in 26 of 30 trials silently dropped a sibling function the description didn't mention. Those were the trials that looked most like success. A partial map was more dangerous than none, because it invited the agent to write code it couldn't check.

One more correction belongs in the record. The first scoring reported 29 out of 30 for one arm. The single failure turned out to be the scoring harness, which read only the first code block in each response and dropped a second file. The raw responses had been kept, so the error was found and re-scored without another run, and every number above is the corrected one. Keep the raw responses.

What this means for a team using agents

The failure is a retrieval problem, and the fix is about reach. An agent makes a cross-cutting change correctly when every site it needs to change is something it can read. It makes it half-correctly, with total confidence, when one of them isn't.

Make the sites findable. A structural model derived from the code at a named revision, and queried during the change, can point an agent at the second total and hand it the source. A hand-written architecture document can do the same until it goes stale, and then it points at code that has moved. Derive the map from the code, and fail the build when the committed map stops describing it.

Let only agreed structure give instructions. How often a pattern appears says nothing about whether anyone agreed to it. The coherence ratchet labels everything it tells an agent as observed, candidate or ratified, and only ratified intent, with an owner and a date against it, is allowed to be phrased as an instruction.

Review for the site that isn't in the diff. A reviewer reading a cross-cutting change should ask where else this rule is computed, and expect the tooling to answer. The diff shows what changed; the question is what should have changed and didn't.

And hold the line on structure. Even with all of that in place, some changes will add a second way of computing something. A ratchet baselined on today's code fails the build when a change makes the structure measurably worse, and turns the exception into a recorded decision with an owner and a review date.

Where the evidence stops

This is one fixture, in one language, with two models, at thirty trials per cell. The confidence intervals don't overlap, and the result replicated across vendors, but it measures a mechanism and not a rate you should expect in your own codebase. The agents also had no search tools. One that could grep the repository might well find the second site, and naming a site alongside a retrieval tool the agent can call is the case still untested. Real systems are messier in the other direction too: the second site might sit in a service the agent can't reach at all. The experiment, the raw responses and the corrected scoring are all in the coherence ratchet repository for anyone who wants to run it on their own terms.