Forecasting, Metrics and the Lies that a Single Number Tell Us

We’ve previously seen that our mental models often make false assumptions due to cognitive bias. Giving a Single number in a report has a similar problem.

The classic Agile number is ‘Velocity’: the average rate at which your team delivers work. It is often shared as just one number. E.g. our team achieves 20 pts a Sprint.

You’re walking and come to a river – there is a sign that says the average depth is 1 foot. Can you safely walk across? You don’t know. For most of its width, the river may be only 6 inches deep, but for a critical 1 foot stretch, it’s 6 feet deep. Not safe to cross.

You’re told the average grade on a test in your class is 75%. What does this tell you about your own mark? Nothing.

The problem with reporting velocity as a single number is that it implies the average number will continue into the future at the same rate. It implies that there will never be any variance.

Consider reporting a range^[1][2].

• Best three sprints in the past year – this is the highest you’re likely to achieve
• Last three sprints – you’re 50% likely to achieve this number
• Worst three sprints in the past year – the least you’re likely to achieve

Along with projecting a cone of uncertainty in your burndown/burnup/cumulative flow diagram, it can also be used to illustrate which items in the product backlog are more or less likely to be achieved by drawing lines in the product backlog.

If you need a more sophisticated model than these, consider using a Monte Carlo simulation. Larry Maccherone has a Chrome only example: http://lumenize.com/ and code showing how he uses it: https://jsfiddle.net/lmaccherone/j3wh61r7/.

With five Sprints to go, we have a clear picture of where in the product backlog we will likely get.

So instead of forecasting a number, now we’re forecasting which items will be delivered, i.e. we’re forecasting value. This drives a better quality conversation because we can have real discussions about which stories or features to sort into which part of the Product Backlog.

This brings up the final problem with velocity as a number (or even range). The number of Product Backlog Items, Features, or User Stories don’t necessarily correlate with how happy the customer is. A small feature may delight the customer if it’s done well (example: ability to freeze the first row in Excel), while a large feature (example: SmartArt in Word) may not make the customer(s) happy.

So even with ranges and probabilities, the use of numbers can still hide the most important thing: Is the customer happy?