How do you know AI is actually helping? Measure it.
Ask any software company whether AI has made their developers more productive and you will hear yes. Ask how they know, and the conversation usually ends, no matter which number they picked this year: thirty percent, fifty, double. None of them come with a baseline, a method, or any way to tell improvement from wishful thinking. As a company that has been building software for clients for more than twenty years, we decided early on that if AI was going to change how we work, we wanted evidence rather than impressions. Here's what that looks like in practice.
“AI is a tool. You don't blame the hammer. It's like a CNC machine: first you learn to operate it, then it makes the parts,”
A machine you learn to operate is a machine you can measure. This is what our measurement practice looks like right now, including the part nobody warns you about.
What we track
Our measurement starts where it always has, with estimates. Every piece of work gets one before it begins. Subtasks are estimated by component, so each discipline's contribution stays visible separately. Time is tracked against every task in Jira, and anomalies go to retrospectives with a simple question: why did this take two days instead of the one hour we expected?
Around that core we track the signals that estimates alone cannot catch:
- How many polishing iterations a piece of work needs, and how long they take
- Defects per version
- Defects found during acceptance versus after release
- How many issues UAT raises before a release gets accepted
- Test coverage
- The full time from starting work to delivery in production
None of these metrics is exotic, and that is the point. Measuring AI does not require inventing new measurements. It requires taking the old ones seriously, because AI changes where time is spent and where defects come from.
On one fixed-budget project with hard deadlines, the numbers flagged performance falling behind, and the retrospective traced it to how AI was being used in the work. The team adjusted, the metrics recovered, and measurement caught the problem early enough to deliver on time.
Keeping the AI rules where the team can see them
The newer practice is treating our AI setup itself as code. Every team works with what we call a harness: the instructions, agents and quality gates that shape how AI participates in their project. A quality gate can be as mundane as checking character counts in generated text, or as consequential as requiring every AI-suggested technology choice to pass an architecture check before it enters the stack. Rules like these are the difference between output you can trust and output you have to babysit.
Each team now keeps its harness in git, versioned next to the project it serves, where it can be inspected, compared and improved instead of living in someone's head. The version history gives us an honest signal for free. A harness that needs correcting every day is not working yet. One that has stabilized is doing its job.
The part nobody warns you about
We expected the hard problem to be designing the methodology. It was not. The metrics were the easy part, and we did not have to invent anything. The real adjustment was learning to treat a personal working style as something shared and visible rather than private.
On one project, the harness needed fixing almost every day for the first few weeks. The retrospective found the cause: two developers were reviewing AI-generated code in different ways, and neither way had ever been written down.
Once the team added both habits to the harness as rules, the daily fixes stopped within two weeks. The hard part wasn't writing the rules. It was turning a personal habit into something the rest of the team could see.
Measurement is the only way we know to tell the difference between adopting AI and just believing in it.
If you are not measuring anything yet, the first step is not a dashboard. It is deciding, starting today, to write something down as the work happens: an estimate before the task, the actual time after it. You can not look back at what went wrong if nothing was written down while it was happening.
