ctaio.dev Ask AI Subscribe free

AI ROI / Return on Tokens

AI ROI · POV

Return on Tokens.

The first productivity proxy handed to you by the party paid per unit.

Engineering leadership has reached for a countable proxy in every era: hours, then story points, then DORA metrics, now tokens. The reaching is not the mistake. What changed with tokens is who supplies the instrument. Hours were counted by the employer, points by the team, DORA by the delivery pipeline. Token consumption is counted, framed and dashboarded by a vendor whose revenue is the quantity displayed. This is the mechanism behind why "return on tokens" cannot answer the question it is being asked, and what to put in the denominator instead.

Return on Tokens: the first engineering productivity proxy supplied by the party paid per unit

Key takeaways

  • Tokens are an input, and the meter belongs to the seller. The count is accurate. What is unreliable is the implied direction: the dashboard's "up" is the vendor's outcome, not yours.
  • The metric degrades in both directions. Pushed up it produces tokenmaxxing; pushed down it steers work away from long-context tasks, which are the expensive ones and often the valuable ones. A real measure improves in one direction. A proxy rots in both.
  • The fix is a denominator, not a better numerator. Cost per completed unit of work, with human review time included. Track token spend as a cost line. Do not promote a cost line to a performance metric.

Why leadership keeps reaching for a countable proxy

The pattern John Cutler names in TBM 437 is that tokens, hours and story points are all the same move: an unmeasurable thing (did this work create value?) gets swapped for a measurable thing standing next to it. The swap is not stupidity. Value is genuinely hard to attribute in software, boards genuinely need a number, and a number that arrives automatically every month will beat a number that requires judgment every time.

So the interesting question is not "is this a proxy" — of course it is — but what makes this proxy different from the last three. Line them up by who holds the meter and the answer is immediate.

FOUR PROXIES, FOUR METERS

Who counted, and what they wanted the number to do

Each generation of engineering productivity metric was measured by a different party. Read the fourth row against the first three.

01

Billable hours

Meter held by: The employer

Wanted the number DOWN per unit of output. Misaligned with the worker, aligned with the buyer.

02

Story points

Meter held by: The delivery team itself

Wanted the number STABLE and predictable. Gamed internally, but the gaming was visible to the managers reading it.

03

DORA metrics

Meter held by: The delivery pipeline

No commercial stake at all. The pipeline does not profit from more deploys. This is why DORA survived scrutiny that the earlier two did not.

04

Token consumption

Meter held by: The model vendor

Revenue IS the number. The first productivity proxy in the lineage supplied by a party paid per unit of it.

This is not an accusation of dishonesty

Nothing above requires a vendor to be acting in bad faith. Usage dashboards exist because customers need to see consumption to forecast a bill, and they are built well for that. The problem is what happens when an artefact designed for billing gets adopted as an artefact for performance management, unchanged and unquestioned, because it was already there and already had a graph on it.

A billing instrument and a performance instrument want opposite things from the same line. The billing view is neutral about direction. The management view that borrowed it inherits a default framing where a rising line reads as progress, and nobody involved had to say so out loud for that framing to arrive. That is the mechanism. It is structural, not moral, which is exactly why it survives being pointed out.

The tell: it rots in both directions

Most bad metrics fail one way. You can usually say "we over-optimised for X and under-invested in Y" and the fix is to bound X. Token spend does not behave like that, and the asymmetry is worth sitting with because it is the strongest single argument against the metric.

Pushed up, under adoption pressure, you get the behaviour this site named tokenmaxxing: prompts padded with context nobody needed, agent loops left to grind on tasks they will not finish, consumption performed because consumption is what is being watched. The dashboard improves. Nothing ships faster.

Pushed down, under cost pressure, the failure is quieter and worse. Long-context work — reading a whole subsystem before changing it, migrating a legacy module, reviewing a large diff — is where token spend concentrates, and it is frequently where the value concentrates too. A blanket instruction to reduce token spend does not reduce waste preferentially; it reduces whatever costs most, which steers usage toward short, cheap, low-stakes prompts. Spend falls. The line looks healthy. The work that justified buying the tool is the work that stopped.

A measure that gets worse whichever way you push it is not a measure that needs tuning. It is a proxy that needs a denominator.

What to measure instead

Keep the token number. It is a real cost and it belongs on the cost line, forecast and budgeted like any other consumption. What it must not be is the performance metric, and the repair is not a cleverer numerator. It is supplying the denominator that was dropped.

Pick a unit of completed work your organisation already counts and already trusts, so you are not also litigating a new definition: a pull request merged after surviving review, a support ticket resolved without reopening, a migration shipped. Then measure the fully loaded cost of producing one of them, and include the human review time the AI output generated. That last inclusion is the one that gets skipped and it is the one that decides the answer, because AI-assisted work often moves effort from writing to reviewing rather than removing it. A cost-per-unit that ignores review time will show a saving that the calendar does not.

Three properties make that denominator work where token count does not. It is counted by you rather than by a supplier. It improves in exactly one direction, so there is no ambiguity about what good looks like. And it cannot be inflated without doing the actual work, which is the only durable defence any metric has against Goodhart.

If you want the arithmetic rather than the argument, the LLM cost calculator handles the numerator and the AI ROI calculator takes the full model. For the team-level version of the same problem, AI team metrics covers which traditional measures break on probabilistic systems, and DORA alternatives covers what to do when the delivery-flow metrics stop describing the work.

Return on Tokens FAQ

What is "return on tokens" and why is it a bad metric?
Return on tokens is the practice of judging AI investment by token consumption: tokens burned per engineer, per team, per month, usually read off a model vendor's usage dashboard. It fails as a metric for a specific structural reason rather than a vague one. Tokens are an input, and the party supplying the dashboard is paid per unit of that input. Every previous engineering productivity proxy was measured by a party with no revenue stake in the number going up: hours by the employer, story points by the team, DORA metrics by the delivery pipeline. Token counts are the first one sold to you by the vendor whose revenue is the metric. The number itself is accurate. What is unreliable is the direction it is presented as wanting to go.
Is high token usage a sign of AI adoption?
It is a sign of AI usage, which is not the same thing. Token volume rises when people use models well and it also rises when an agent loops on a task it cannot complete, when a prompt carries an entire repository it did not need, and when someone has worked out that consumption is being watched. That last case has a name coined on this site: tokenmaxxing, the deliberate inflation of token counts once they become the reported metric. Because usage and waste both move the number in the same direction, the number cannot distinguish between them. A team that halves its token spend by learning to scope context tightly will look, on the dashboard, exactly like a team that stopped using AI.
What should engineering leaders measure instead of token spend?
Cost per completed unit of work, with the token spend as the numerator and a delivered outcome as the denominator. The denominator is the whole point and it is the part that gets dropped. Pick a unit your organisation already counts and already trusts: a merged pull request that survived review, a resolved support ticket, a shipped migration. Then track the fully loaded cost of producing one, including the human review time the AI output created. Token spend on its own answers "how much are we consuming"; cost per completed unit answers "is this cheaper than it was", which is the question the CFO actually asked. Track token spend as a cost line, by all means. Just do not promote a cost line to a performance metric.
Why does optimising for token spend fail in both directions?
Because the metric is bad whether you push it up or down, which is the clearest available tell that it is a proxy rather than a measure. Push it up, under adoption pressure, and you get verbose prompting, agent loops left running, and consumption performed for the dashboard. Push it down, under cost pressure, and something worse happens quietly: long-context work is the most expensive kind of token spend and often the highest-value, so a cost-cutting mandate steers usage toward short, cheap, low-value tasks while the work that justified the investment goes untouched. Spend falls, the dashboard improves, and the return disappears. A real measure gets better in one direction. A proxy degrades in both.
Is this just Goodhart's Law applied to AI?
Goodhart's Law is the general case and it applies, but it does not explain what is specifically new here. Goodhart says a measure that becomes a target ceases to be a good measure, and that dynamic hit hours and story points too. The new element is the conflict of interest in who supplies the instrument. Story points were gamed by the team that reported them, which at least kept the gaming inside the organisation and visible to its managers. Token dashboards are built, framed and defaulted by a supplier whose revenue is the quantity displayed. That does not make the vendor dishonest. It makes the dashboard a sales instrument as well as a management one, and those two purposes recommend opposite directions.

Continue the AI ROI cluster

The proxy is the argument. The calculators do the math.

Thomas Prommer
Thomas Prommer Technology Executive — CTO/CIO/CTAIO

These salary reports are built on firsthand hiring experience across 20+ years of engineering leadership (adidas, $9B platform, 500+ engineers) and a proprietary network of 200+ executive recruiters and headhunters who share placement data with us directly. As a top-1% expert on institutional investor networks, I've conducted 200+ technical due diligence consultations for PE/VC firms including Blackstone, Bain Capital, and Berenberg — work that requires current, accurate compensation benchmarks across every seniority level. Our team cross-references recruiter data with BLS statistics, job board salary disclosures, and executive compensation surveys to produce ranges you can actually negotiate with.