Engineering Predictability Metrics: Say/Do Ratio, Forecast Accuracy, and When to Stop Measuring Them

What are engineering predictability metrics?

They measure the gap between what an engineering team committed to and what it delivered, in scope and in time. Every metric in this family has the same shape: a promise recorded at time T, an observation at time T+n, and a ratio or error between them. That shape is the source of their usefulness and of their main weakness. The team that makes the promise also controls how big the promise is.

Predictability is a property executives feel more than they measure. The CEO remembers the two launches that slipped, not the eleven that landed. I covered this in what CEOs expect from their CTO: when you say Q2, it ships in Q2. Metrics exist to replace that memory with a number, so the conversation is about a trend and not about the last miss.

Why is predictability in tension with outcomes?

Because a team optimizing for hitting dates will avoid the work most likely to move outcomes, which is also the work most likely to surprise it. Marty Cagan made this case in Predictability vs Outcomes (SVPG, October 2026). His argument: the familiar output-versus-outcome debate is a symptom. "The real trade-off is the relative priority of outcomes versus predictability," and in many companies predictability wins by default, with outcomes assumed to follow from shipping what was promised.

The mechanism is simple. Committing to outcomes means accepting that many roadmap ideas will not work, which Cagan says plainly: "many, if not most, of the product ideas on your roadmaps are simply not going to work." A roadmap of dated features cannot absorb that. When an idea fails in discovery, the predictable move is to ship it anyway because the date was promised. The say/do ratio stays at 100 percent and the product gets worse.

Cagan does not call predictability bad. He calls it something some situations genuinely require, handled with specific techniques, inside an organization that otherwise prioritizes outcomes. That framing is the one I use below: predictability metrics are an instrument for specific commitments, not a general health score for the engineering org.

Which predictability metrics should you track?

Fewer than you think. The table lists the common ones with formula, the healthy range where a source actually publishes one, and how each gets gamed. Where there is no sourced benchmark, the column says so. Do not borrow a number from a vendor deck and turn it into a target.

If I could keep only two, I would keep cycle time percentiles and forecast calibration. Both are computed from data the team does not directly negotiate, and both feed real forecasts. Say/do ratio is the most popular and the most fragile. Faros puts it well: it shows how good the team is at estimating its capacity, not whether it is working on the right things.

How do you calculate say/do ratio without breaking it?

Divide the work completed in the cycle by the work committed at the start of it, and multiply by 100. The arithmetic is trivial. The definitions are where it breaks.

• Freeze the commitment. Snapshot the committed set at the end of planning. Items added mid-sprint go in the unplanned bucket and never in the denominator.
• Pick one "done". Deployed to production behind a flag is a reasonable default. If "done" drifts between merged, deployed and released, the ratio drifts with it.
• Count items, not points, once velocity becomes political. Points are a negotiation. Item counts for similarly sliced work are harder to inflate.
• Report it at team level only. Say/do aggregated across teams, or compared between teams, rewards whichever team commits most conservatively.
• Read it with the unplanned work share. A 60% say/do with 45% unplanned work is an interruptions problem, not a planning problem.

GitLab once set a 100% say/do ratio as a quarterly key result (anything labelled Deliverable should ship), and the public issue records 64% for that quarter. Even a disciplined, transparent organization finds 100% hard to sustain. A team that hits it every time is almost certainly committing below capacity.

How should you forecast instead of estimating?

Forecast from throughput history and give a date with a probability attached. Daniel Vacanti's When Will It Be Done? (2018) lays out the Monte Carlo approach: sample the team's historical throughput many times to simulate how long the remaining items will take, then read off dates at the 50th, 85th and 95th percentiles. The output sounds like "85% likely by 14 March", which is a sentence a CFO can plan with.

This changes what predictability means. A team does not become predictable by estimating better. It becomes predictable by narrowing the spread of its cycle times: smaller slices, less work in progress, fewer items waiting on review. Forecast accuracy then follows from flow, and you stop asking engineers to guess. That is also why the cycle time spread row in the table matters more than say/do.

If your team now ships a lot of agent-written code, re-baseline. Implementation got faster, verification did not, and history from before the tooling change will make your forecasts optimistic. I covered the measurement side of that shift in AI team metrics.

When does predictability actually matter?

When a date has a cost attached that does not depend on what you ship. That is a short list, and it is worth writing down so the rest of the roadmap is not held to the same standard.

For the first list, Cagan's answer in Empowered (Cagan and Jones, 2020) is the high-integrity commitment: do enough discovery first to know the solution works and roughly what it costs, then commit to the date, and keep these commitments rare. Measure milestone hit rate on that small set. Holding a 40-item roadmap to the same standard just teaches teams to stop putting ambitious items on it.

Why does it take two quarters to know whether a team is predictable?

Because predictability is a lagging indicator, and the sample is small. A team running two-week sprints produces six data points per quarter. Two of those will be distorted by holidays, an incident or a reorg. One quarter of say/do data tells you very little, and by the time you have enough to see a trend, the conditions that produced it have changed.

The lag is worse for the metric executives care about most. A quarterly roadmap commitment either lands or misses once per quarter. Four data points per year per team is not a measurement, it is an anecdote with a spreadsheet. You will know whether a new team is reliably predictable after about two quarters at the earliest, and by then the expensive misses have already happened.

Two practical consequences follow. First, lean on leading inputs that update weekly: work in progress, age of in-flight items, review wait time, unplanned work share. Those tell you this week that next month's forecast is in trouble. Second, do not change the measurement when a new leader arrives. Every redefinition of "done" or "committed" resets the clock to zero and buys you another two quarters of not knowing. The same lag applies to board-level engineering metrics, where the gap between cause and visible effect is even longer.

When should you stop measuring predictability?

When the number has stopped changing and stopped informing decisions. Predictability metrics are a training tool. They are valuable while a team learns to slice work, protect focus and say no to mid-sprint additions. Once say/do has held steady for several quarters and forecasts land inside their windows, continuing to report the metric weekly has a cost: people start protecting the number.

• Retire say/do from dashboards once it is stable. Look at it in retros if something feels off.
• Keep cycle time percentiles permanently. They are the input to every forecast and cost almost nothing to maintain.
• Keep milestone hit rate only for high-integrity commitments, and report it next to the number of commitments made.
• Move review time to outcomes: the adoption, retention or revenue metric each initiative was supposed to move.
• Never put predictability metrics in individual performance reviews. That is how they turn into sandbagging contests.

This mirrors the advice in Beyond DORA: delivery metrics tell you whether the machine runs, not whether it is pointed at anything worth reaching. If your goal-setting framework is what keeps pulling the conversation back to dates, the OKR alternatives guide covers frameworks that track outcomes without quarterly theater, and the engineering roadmap template shows how to separate the few dated commitments from everything else.