The work in practice
The evaluation question changed from “is the tool good?” to “what happens to the work around it?” Pilots were judged on review load, quality, end-to-end flow and adoption, not only on how fast one task was done.
A faster task in an unchanged system
The early results were encouraging and familiar: people finished parts of their work faster. The risk was treating that as the answer. A change written faster still waits for review, approval and a product decision. And time saved is not value until someone decides what it is for.
Where to start
I recommended starting where delivery was actually constrained and where the value to the business was highest, which often meant getting changes to customers sooner. The easiest task to speed up is rarely the one holding the work back.
How I ran the evaluation
I judged tools against the work they had to support, not against feature lists. Each pilot stated what was measured, who was involved and over what period, so a result could be reproduced or challenged.
I assessed productivity by measuring flow end to end: cycle time, release frequency and defect counts. Faster completion of an individual task was only part of that picture.
Then we looked past the task. Did code review take more effort or more waiting? Did quality hold, or did rework rise? Did changes reach release sooner, end to end? Did people keep using the tool once the novelty passed? My recommendations rested on that wider picture.
Cost was part of the same picture. I modelled usage and consumption alongside licence prices, because what a tool costs depends on how heavily it is used, and that is hard to see from the price list.
Where the saving went
In one pilot, the time saved in writing code reappeared in review. Reviewers spent longer checking the changes. On a task-speed measure the pilot looked like a clear win; across the delivery flow, much of the effort had simply moved.
That changed the recommendation. Fix the review step first, then put the released capacity into backlog work and into quality, rather than counting it as a saving nobody would see.
Controls before scale
Wider use needed rules that did not exist for hand-written code. Together with colleagues, I co-designed them: how AI-generated changes are reviewed, what AI tools and agents may access or do and where a person must approve, and which data and code may be shared with a tool at all.
Changing roles, not just tools
The work changed more than the toolset. I led the changes to review and verification roles, training on working with AI tools, and the ways of working in teams whose workflow now includes AI. Reviewing a change you did not watch take shape is a different skill, and it needed to be treated as one.
Why I built part of it myself
Writing part of the implementation showed me where AI output adds verification work and where it genuinely removes it. A vendor comparison would not have shown that.
What changed my approach
AI adoption is mostly an operating-model change with a tool attached. The tool is the smaller decision. The larger ones are where to apply it, what controls it needs and what the organisation does with the capacity it releases, which the accompanying essay works through.
Related work and reading