Coaching AI adoption from how your team actually uses it

By Stephan Swart · · 7 min read

Claude Code usage analytics show what people do in their sessions, which a survey cannot. I capture tool use, skills and stuck turns, with the team told first, then coach the real gaps while the engagement is still running.

Ask a team how they use Claude Code and you get a reasonable answer. They plan before big changes, they keep sessions short, they check the output. Then you look at their real sessions on a deadline and find routine git chores typed into the main conversation, one conversation running all day across unrelated jobs, and nearly everything on the most expensive model.

Neither picture is a lie. People describe the habit they believe they have. Coaching works best when I can see both pictures, so on most engagements I set up Claude Code usage analytics alongside the training and use what they show to steer it.

What surveys and self-ratings can tell you

I still run a tracker before the first masterclass and after the last. The tracker asks situation questions with one best answer, plus a few confidence ratings, and it tells me a lot about judgement and understanding. I explain how that works in how to measure AI adoption in a software team.

What a survey can't show is how people work on a Tuesday afternoon under deadline. Self-ratings describe intentions. When a team is busy, the habits that slip are the ones people would still tick as "always do this" on a form.

What session capture shows

Claude Code can report what happens in each session: every prompt, every tool it runs, whether that tool worked, and how long each request took. Grouped by request, those records turn into four signals I find useful.

  • Tool mix and failure rate. Which tools people reach for, and how often those tools fail on them. Reaching for the right tool and using it wrong is the most teachable case there is.
  • Skill coverage. Which skills and commands each person has discovered, and which ones their teammates use every day that they have never touched.
  • Stuck turns. One request that cost twenty tool calls and forty minutes. The worst of these are my shortlist of sessions worth reading closely.
  • Permission friction. Tools people keep refusing, which usually means they don't trust them yet or have them configured wrong.

None of these four signals is a score. The signals tell me where to look.

How I set it up

There is no new software on anyone's laptop. Claude Code has OpenTelemetry built in, and Anthropic's monitoring documentation describes how an administrator can switch it on for everyone through the organisation's managed settings. That means nobody edits a config file, and nobody gets left out because they missed a setup step.

The data goes to a small dashboard I run for the engagement. The dashboard has three layers, and each layer is a separate decision:

  1. Usage counts. Tokens per person per day. Useful for spotting cost patterns, and the least interesting layer for coaching.
  2. Session events. Which tools ran, which failed, which skills were used, how long each request took. Session events give every signal above without storing a word anyone typed.
  3. Content capture. The text of prompts and Claude's replies, so I can read a transcript. Content capture is its own switch, and it stays off until the team has been told.

Prompt text is redacted by default in Claude Code itself, so content capture is something you have to choose on purpose. On my side, conversation text is deleted after a fixed retention period, 90 days in the setup I use.

Recording conversation content is a different thing from counting tokens, and I treat the two differently.

  • The team hears about it before it starts. I explain what is captured, who can see it, how long it is kept and what it is for: aiming the training at real gaps. Content capture goes on only after that conversation.
  • Everyone can see their own data. Each person has a page with their own activity, skills and transcripts. Anything I can read about them, they can read too.
  • Group reports never name anyone. Findings are written as "three of five people" or "four of five teams". When a group lead downloads an export, people appear as labels with no names or emails attached.
  • The client can rule things out. At the JustSolve hackathon, the client decided that prompts typed in the evening and overnight were off limits. Those turns were removed before anyone, human or Claude, read a transcript.
  • Secrets are flagged and never repeated. When a transcript contains something sensitive, such as an access token pasted into a prompt, the report names where it is and recommends rotating it. It never quotes it.
  • The dashboard never calls an AI model. The dashboard only stores and displays data. The qualitative reading is a separate step I run myself on exported transcripts, with the exclusions above already applied.

Reading the stuck turns, myself and with Claude

The numbers produce candidate problems. The transcripts decide which ones are real.

I read the worst stuck turns myself, and I hand a set of exported transcripts to Claude with a skill I built for the job. The skill starts from the numbers, writes down candidate patterns, then reads the full transcripts to confirm or drop each one. The skill looks for repeated instructions that belong in a project's CLAUDE.md, moments where a feature people hadn't discovered would have saved the turn, and how people recover when Claude gets something wrong. The skill also has to find what the team does well, held to the same standard of evidence.

Reading matters because raw numbers mislead. A well-planned fan-out of five subagents can look like one request that made 500 tool calls. Results that subagents hand back arrive looking like typed prompts, and in one session that meant 63 "prompts" when the person had typed about 25. If I coached from the raw counts, I'd be telling the most skilled people they were stuck.

Token counts need the same care. A huge number points at a habit, and it's no achievement in itself. On the first day of the JustSolve hackathon, the heaviest single sessions had re-read between 50 and 140 million input tokens. That was the cost of one conversation running all day across unrelated jobs. I explain why volume measures so little in the measurement article.

Feeding it back while the engagement is still running

A report that arrives after the programme has ended is a post-mortem. I want findings in front of the team while they can still change something.

The JustSolve hackathon is the clearest example. Five teams who had finished the full masterclass spent three days building. On the afternoon of day one, each team got a short report on its own usage, plus a cross-team summary of what the teams had in common. Every habit in those reports had been taught, and pressure had pushed the habits aside:

  • In all five teams, routine chores like pulling, pushing and opening a pull request were typed into the main working conversation, each one a full model turn.
  • In all five teams, someone kept one conversation open all day across unrelated jobs.
  • Four teams ran almost everything on the largest model, and almost nobody checked their usage.

Each finding came with the principle behind it, a better habit with a prompt to copy, and what should look different in the next export.

The day two report showed every team had acted on its report within hours. Two teams turned their git chores into shared skills. Two moved most of their new sessions to a smaller model. In one team, three of four people now checked their usage, up from almost none. New problems had appeared too. Four teams now ran several Claude sessions in parallel in the same folder, and the sessions kept tripping over each other. The parallel-session problem went into the day two report with a fix: give each session its own git worktree.

Over a longer engagement the feedback is slower and the steps are the same. Between sessions I look at what the group is doing. A pattern across the group reshapes the next session. A problem one or two people keep hitting goes to office hours, which I hold between every masterclass. An instruction people keep typing becomes a line in CLAUDE.md or a shared skill. The next export tells me whether it worked.

Feeding findings back is the part of AI adoption after the licences that leaders rarely see. The training adjusts to how the team actually works, week by week, and the evidence for each adjustment is something you can read.

See what your team is really doing

If you want AI adoption coaching aimed at the habits your team actually has, book a conversation. I'll explain how usage capture fits into the programme and what your team would need to agree before it goes on.

Keep reading