In defence of the token-spend metric: AI adoption vs delivery
You have probably seen the memes, or read the jokes, about how absurd it is to measure the number of tokens a company’s employees spend. I want to stir the pot a little and challenge the idea that this metric is completely useless.
Why do we need metrics at all? To measure and control a process, especially one that is still in early adoption. It’s fairly obvious that a good metric should measure the value a process creates (outcome, not output). From that angle, token spend really does look absurd. It’s entirely unclear what the tokens were spent on and whether they brought any benefit. You can easily become the champion of this metric by breaking the context-caching practices that the whole economics of the agentic loop rests on.
Let’s look at introducing AI into an organisation from a slightly different angle and split it into two stages:
- Adoption. Getting employees broadly familiar with AI tools and starting to use them in practice.
- Delivery. Evaluating how AI is used and tuning it to get maximum value for the organisation.
Why do we need these stages? Because new practices can be introduced either in depth or in breadth. In depth means we carefully set up the practice in one team or project, then a second, a third, and then scale the experience across the organisation. In breadth means we launch the new practice as widely as possible right away and then tune it empirically based on what we learn.
The first approach is cheaper and more reliable, but takes much longer. The second is much faster, but significantly more expensive and riskier.
With AI, the main factor is time. Everyone is afraid of being late and falling behind the global revolution. So only small organisations can afford to go “in depth”. Large ones are forced to go “in breadth”. And then our two stages become very obvious, and each of them needs its own metrics for measurement and control.
For the adoption stage, it’s important to know that maximum breadth has been reached, so that in the second stage nobody can say the change simply hasn’t reached them yet. That’s why token spend fits this stage perfectly. It lets you:
- Find those who resist the new reality and aren’t ready to adapt.
- See who isn’t willing to invest in their own learning and uses AI tools inefficiently.
- Find potential islands of innovation and examine them for the value they create.
- Take early control of mass LLM API usage and build efficient infrastructure.
- Shake up the organisation a little with a global metric. This matters especially for slow enterprise organisations.
Expensive? Definitely! Open to different interpretations? It will be! But it’s as fast as possible and covers the whole breadth of the organisation.
Then the delivery stage begins, where you can take specific people, teams and projects and tune how they use AI to get maximum value. But that’s a completely different story, with its own metrics…