Sooner or later the board asks what the AI licences have changed. The easy answer is a dashboard: seats activated, prompts sent, tokens used, lines generated. Those numbers are easy to collect and they go up and to the right, yet they tell you almost nothing about whether the work got better.
Here is how I measure AI adoption in software teams instead, and what I've learned about the traps along the way.
Activity counts measure volume
When I help a client run a hackathon at the end of a programme, the participant guide includes a list of things that are explicitly not achievements:
- The highest token usage
- The largest number of prompts
- The most MCP servers or tools
- The most generated code
- The most commits
- Checking every SDLC activity without meaningful depth
- A polished presentation that hides a shallow learning journey
The same thinking applies to measuring a whole organisation. "We used four million tokens and made 212 commits" is a count, and a count proves nothing about learning. A team can produce enormous volume while quietly redoing half of it.
Self-reported productivity has the same weakness. In the 2025 DORA report, 90% of respondents said they use AI at work and more than 80% believe it has increased their productivity. The same research found that AI adoption still has a negative relationship with software delivery stability. People feeling more productive and the system staying stable are two separate questions, and you need to measure both.
Ask situation questions with a best answer
The core of my measurement is a short tracker everyone fills in twice: once before the first masterclass and once after the last. The questions are identical both times, so the difference between the two runs is the result. The tracker takes about twelve minutes.
Most of the tracker is ten situations, each with one best answer. For example:
An agent reports: "I have implemented the change and everything works." What is the most useful next question?
The options include "Are you sure?", "Can you explain your reasoning?" and "Please double-check your work." The best answer is "What did you run, and what was the output?" Every wrong option is something a competent person actually does, so nobody can pick the right one by elimination.
A few rules make these questions useful:
- Describe the situation in plain words. No tool names, commands or pattern names. If a question can only be answered by recognising a term from the course, that question measures vocabulary. Every question has to be answerable by someone who has never opened a terminal, so a designer and a developer are measured on the same footing.
- Always offer "I'm not sure". That option scores zero, and its rate is often the most telling number. When that rate falls from the first run to the last, people have stopped guessing.
- Expect people to get the first run wrong. The first run happens before anything has been taught. I tell participants so up front, and I tell them the results point at the trainer as much as at them. If a topic doesn't move across the room, that session needs rebuilding.
Report several numbers, never one total
I group the ten situations into three tiers. Some can be solved by good engineering judgement. Some need the specific insight the programme teaches. Two are deliberately easy, so I can spot anyone clicking through without reading. Averaging those tiers into one score cancels them against each other and produces a meaningless middle, so I report them separately.
Alongside them I report the "I'm not sure" rate and how much it fell, and the gap between how confident people feel and how they actually answered.
Expect a dip in confidence
The tracker also asks people to rate their confidence on three capabilities, such as telling a real "it works" from an unchecked claim by asking for the right evidence.
On the second run, some people rate themselves lower. A lower rating looks like failure. Usually a lower rating means they now understand how much the capability involves. When confidence dips on something the group clearly practised, I read it as recalibration and say so in the client report before anyone asks.
Separate the person from the organisation
Part of the tracker asks how people feel about working this way. Two of those questions are about the employer:
- I have time to work this way while hitting my current commitments.
- The way my work is measured rewards working this way.
I keep those two questions out of the personal score. Folding them in would label someone who believes in the approach but works under a bad incentive system as a sceptic. The question about how work is measured rarely moves in a matter of weeks, because nobody rewrote the performance review. When that answer stays low, it tells the sponsor the blocker is how the organisation measures its people.
I also ask about behaviour directly. How often in the last two weeks did a change, spec or design an agent produced go out with only your review between it and done? The last time an agent got something wrong twice in a row, did you change what you gave it, or take the task back and finish it by hand? Those answers describe habits more reliably than any opinion question.
Judge the change from a tagged baseline
Survey answers need corroborating with real work. The strongest evidence comes from the hackathon at the end of the programme.
Before the start, each team tags its starting point in git and writes a baseline declaration of two pages at most. Judges score only the change from that tag to a frozen judging branch. Teams may bring existing code, designs and a product requirements document, and those give the judges context, but only the increment is judged.
In a recent client hackathon, nearly half the score, 45 of 100 points, went to learning and AI use across the lifecycle. A collaboration multiplier from 0.85 to 1.05 scaled the whole score, based on whether the presentation showed who did what. Each team told an evidence story in six parts: problem, experiment, AI approach, evidence, result, learning. The strong version of that evidence is the failing test, the prompt that fixed it, and the passing run, saved together.
Koketso, a developer, summed up what that kind of evidence captures:
"My way of planning and execution changed because of multi agents, workflows, pipelines, and artefacts."
Keep delivery metrics in view
Capability and behaviour are leading signals. Delivery is where the effect should eventually show.
At a software company where I've been fractional CTO, I run an engineering department of five cross-functional teams, and the numbers I point to there are delivery numbers. Adopting Agile, Lean and DevOps principles reduced the software delivery cycle by 75% and increased deployment frequency by 400%, and all teams now average 90% unit test coverage. Delivery numbers like those are what I'd want next to any AI adoption effort: cycle time, deployment frequency, and how often changes fail. Given the DORA finding on stability, watch how often changes fail most closely.
What this can and can't tell you
A pre and post tracker on one group, with no control group, can't separate the programme from weeks of ordinary practice, and the first run itself draws attention to the topics it asks about. I say so plainly in every report. That limit is why the corroborating evidence matters: the written team agreements, the executable specifications, the working slice, and the judged increment.
Privacy matters too. Individual answers stay with me. The client sees how the group moved and nothing else, and I never report a role or team split smaller than five people, because the one designer in a chart is identifiable even without a name.
If you want the wider picture of what has to change around the measurement, start with AI adoption after the licences.
Talk through what you need to show
If your board is asking what the AI tools have changed and you don't yet have an answer you trust, book a conversation. I'll look at where your teams are and tell you what the programme can measure for you.