Software engineering estimates are garbage
That’s not because companies are using the wrong methods or tools. Work-breakdown structure or analogy-based? Mechanical or judgmental combination? Function, use case, or story points? SEER-SEM, WMFP, or Wideband Delphi? Fine.
The tools aren’t the problem. Rather, most estimates are garbage because they’re based on a fundamentally flawed understanding of how quality software is built.
The impact goes far beyond cost overruns and missed deadlines. The typical approach to estimates ends up forcing bad behavior while privileging vanity metrics over delivering actual business value.
In Agile environments, estimates are often based on story points and velocity. How “complex” will it be to create a discrete piece of the solution? And how long does it typically take us to complete a story of that complexity? (I’ve written previously about how this approach to Agile corrupts Scrum with Waterfall methods of control.)
When estimating this way, we understand that not everything will go according to plan. But underlying most estimates is a dangerous assumption that even this uncertainty can be quantified and factored into our estimates. If optimistic engineers tend to underestimate how long a given task will take by 15%, we just feed that correction into the formula for a better prediction.
This obsession with specifying and measuring the full process in advance wraps a plus or minus variance around a system that views engineers as machines pushing predictable work products through a pipeline at a steady stream. Then this metaphor of software development is treated as real and translated into mathematical calculations that clad fantasy functions with a veneer of quantitative validity.
Yet to state what should be obvious, human beings are not machines. (Thank goodness for that.) And maybe less obviously, the complexity of any non-trivial software engineering task is almost impossible to accurately estimate in advance.
Our field is so new and rapidly changing. This makes last week’s performance a very poor predictor of next week’s velocity. So many of the interesting challenges we face each day are novel and unknown, and even the known ones won’t stay still.
Consider a trivial example: implementing a login page. Any experienced software engineer has done this dozens or hundreds of times. We know the pattern of the solution well, and we can make some predictions about how long the next one will take. But then along comes a new, more secure way of handling authentication and authorization, and suddenly we have to rethink and reimplement how a basic login page will work.
Most software engineering challenges are far more complicated than a login page. This is as it should be when we’re tackling big problems and creating substantial value. We’re doing things that haven’t been done before, or maybe haven’t been done as well as we think is now possible. We’re in uncharted territory, with a compass but no map.
This uncertainty, in other words, is good. It’s a sign that our ambition is sufficiently visionary, that we’re taking on work that is meaningful and worthwhile. Poor predictability is not the problem. Arbitrary estimates are.
As a statistician might put it, there is too much noise in the system, more variance than we could possibly correct for in our estimates. And the work we’re trying to estimate, if it’s work worth doing, is fundamentally non-deterministic.
When estimates are based on the myth of metronomic coding machines tackling deterministic work, they’re a complete waste of time.
But wasting time is only the beginning. It’s the least serious cost.
Garbage estimates don’t account for the humanity of the people doing the work. Worse, they imply that only the system and its processes matter.
This ends up forcing bad behaviors that lead to inferior engineering, loss of talent, and ultimately less valuable solutions.
