AI ROI · POV
Return on Tokens.
The first productivity proxy handed to you by the party paid per unit.
Engineering leadership has reached for a countable proxy in every era: hours, then story points, then DORA metrics, now tokens. The reaching is not the mistake. What changed with tokens is who supplies the instrument. Hours were counted by the employer, points by the team, DORA by the delivery pipeline. Token consumption is counted, framed and dashboarded by a vendor whose revenue is the quantity displayed. This is the mechanism behind why "return on tokens" cannot answer the question it is being asked, and what to put in the denominator instead.
Key takeaways
- Tokens are an input, and the meter belongs to the seller. The count is accurate. What is unreliable is the implied direction: the dashboard's "up" is the vendor's outcome, not yours.
- The metric degrades in both directions. Pushed up it produces tokenmaxxing; pushed down it steers work away from long-context tasks, which are the expensive ones and often the valuable ones. A real measure improves in one direction. A proxy rots in both.
- The fix is a denominator, not a better numerator. Cost per completed unit of work, with human review time included. Track token spend as a cost line. Do not promote a cost line to a performance metric.
Why leadership keeps reaching for a countable proxy
The pattern John Cutler names in TBM 437 is that tokens, hours and story points are all the same move: an unmeasurable thing (did this work create value?) gets swapped for a measurable thing standing next to it. The swap is not stupidity. Value is genuinely hard to attribute in software, boards genuinely need a number, and a number that arrives automatically every month will beat a number that requires judgment every time.
So the interesting question is not "is this a proxy" — of course it is — but what makes this proxy different from the last three. Line them up by who holds the meter and the answer is immediate.
FOUR PROXIES, FOUR METERS
Who counted, and what they wanted the number to do
Each generation of engineering productivity metric was measured by a different party. Read the fourth row against the first three.
Billable hours
Wanted the number DOWN per unit of output. Misaligned with the worker, aligned with the buyer.
Story points
Wanted the number STABLE and predictable. Gamed internally, but the gaming was visible to the managers reading it.
DORA metrics
No commercial stake at all. The pipeline does not profit from more deploys. This is why DORA survived scrutiny that the earlier two did not.
Token consumption
Revenue IS the number. The first productivity proxy in the lineage supplied by a party paid per unit of it.
This is not an accusation of dishonesty
Nothing above requires a vendor to be acting in bad faith. Usage dashboards exist because customers need to see consumption to forecast a bill, and they are built well for that. The problem is what happens when an artefact designed for billing gets adopted as an artefact for performance management, unchanged and unquestioned, because it was already there and already had a graph on it.
A billing instrument and a performance instrument want opposite things from the same line. The billing view is neutral about direction. The management view that borrowed it inherits a default framing where a rising line reads as progress, and nobody involved had to say so out loud for that framing to arrive. That is the mechanism. It is structural, not moral, which is exactly why it survives being pointed out.
The tell: it rots in both directions
Most bad metrics fail one way. You can usually say "we over-optimised for X and under-invested in Y" and the fix is to bound X. Token spend does not behave like that, and the asymmetry is worth sitting with because it is the strongest single argument against the metric.
Pushed up, under adoption pressure, you get the behaviour this site named tokenmaxxing: prompts padded with context nobody needed, agent loops left to grind on tasks they will not finish, consumption performed because consumption is what is being watched. The dashboard improves. Nothing ships faster.
Pushed down, under cost pressure, the failure is quieter and worse. Long-context work — reading a whole subsystem before changing it, migrating a legacy module, reviewing a large diff — is where token spend concentrates, and it is frequently where the value concentrates too. A blanket instruction to reduce token spend does not reduce waste preferentially; it reduces whatever costs most, which steers usage toward short, cheap, low-stakes prompts. Spend falls. The line looks healthy. The work that justified buying the tool is the work that stopped.
A measure that gets worse whichever way you push it is not a measure that needs tuning. It is a proxy that needs a denominator.
What to measure instead
Keep the token number. It is a real cost and it belongs on the cost line, forecast and budgeted like any other consumption. What it must not be is the performance metric, and the repair is not a cleverer numerator. It is supplying the denominator that was dropped.
Pick a unit of completed work your organisation already counts and already trusts, so you are not also litigating a new definition: a pull request merged after surviving review, a support ticket resolved without reopening, a migration shipped. Then measure the fully loaded cost of producing one of them, and include the human review time the AI output generated. That last inclusion is the one that gets skipped and it is the one that decides the answer, because AI-assisted work often moves effort from writing to reviewing rather than removing it. A cost-per-unit that ignores review time will show a saving that the calendar does not.
Three properties make that denominator work where token count does not. It is counted by you rather than by a supplier. It improves in exactly one direction, so there is no ambiguity about what good looks like. And it cannot be inflated without doing the actual work, which is the only durable defence any metric has against Goodhart.
If you want the arithmetic rather than the argument, the LLM cost calculator handles the numerator and the AI ROI calculator takes the full model. For the team-level version of the same problem, AI team metrics covers which traditional measures break on probabilistic systems, and DORA alternatives covers what to do when the delivery-flow metrics stop describing the work.
Return on Tokens FAQ
What is "return on tokens" and why is it a bad metric?
Is high token usage a sign of AI adoption?
What should engineering leaders measure instead of token spend?
Why does optimising for token spend fail in both directions?
Is this just Goodhart's Law applied to AI?
Continue the AI ROI cluster
The proxy is the argument. The calculators do the math.