Why pricing is hard
Pricing and project is hard. But software development projects are even harder.
It's well known that estimating the cost of a software project is very difficult. A study by HBR revealed that one in six IT projects had overruns of more than 200% and were late by almost 70%. That same article also details the Levi Strauss SAP disaster that cost them $192.5 million (in 2008) and cost David Bergen, the chief information officer, his job.
And it turns out that IT projects have higher success than software projects. Another study by McKinsey found that software projects with budgets over $15M went over budget by an average of 66% and schedule overruns averaging 33%.
Unfortunately, it’s common to look at this pattern, see that estimating software project timelines is hard and just … give up. There’s an established “No Estimates” position that says we should entirely stop giving estimates on software projects. Many Agile methodologies involve arbitrary scoring systems – story points, t-shirt sizing, etc. – deliberately designed to help avoid giving estimates in time-scale units.
- Jacob Kaplan-Moss / Co-creator of Django
Here at BoltAffect we're an Agile company, we hate giving estimates, but we do it anyway. But why is it so hard?
The fundamental problem with software estimation is what we call the cone of uncertainty:
At the start of the project when we're trying to say how long something will take and how much it will cost we're short on details. As the work progresses the path to done becomes more clear. But by that time you've already blown through your budget.
In other words, the further away from done that we are, the less we know how long it it will take.
We're tempted to just say "well just double or triple the estimate". But we always run up agains Hofstadter's Law:
It always takes longer than you expect, even when you take into account Hofstadter’s Law.
- Douglas Hofstadter
Hofstadter published his law in 1979 specifically addressing the difficulty in accurately estimating the time it takes to complete tasks of substantial complexity.
And complexity is the heart of the problem. The cone of uncertainty exists because the problem is complex.
Solution #1
What we prefer to do here at BoltAffect is reverse engineer a budget. Instead of “how long will Feature X take?” we convert the budget to development days and ask “what can we get done in the next two weeks?”
This often is a MUCH better way of thinking about software projects, especially long-lived ones. And it keeps us focused on constantly releasing software with business value. We aren't just spending the budget, we're aiming to release software within the timeframe that the budget allows.
This is also the most cost effective way of planning since there's no extra work that's done to create the estimate.
Solution #2
Monte Carlo simulations!
Just like other complex issues of life with lots of uncertainty (think retirement planning), we run Monte Carlo simulations to give you an estimate for your project. This process looks like the following:
- Break down work into less-complex tasks
- Estimate the complexity of each task
- Estimate the uncertainty of each task (this is the secret sauce)
- Do the math to get expected- and worst-case estimates
- Refine, if needed
Our Monte Carlo simulations can be very good and we have had business owners tell us how helpful these are for decision making. In fact, a few years ago we had a project that was estimated to be $100,000 (not by us). But our simulation showed that the likelihood of actually hitting $500,000 was pretty high and their budget was closer to $100K. In this case the risk exceeded the client's risk tolerance. We created a de-risked project plan that would fit in budget but they felt the smaller feature set didn't deliver enough value.
Sadly for us, that business owner decided not to move forward with the project. And in the end I think they were relieved that they didn't make a costly mistake that handicapped their organization.
Of course, most of the time software is well worth the investment. And we recommend an 18 month ROI (but I digress).
Creating an estimate of this caliber, like a retirement plan, is time consuming. So if you decide this is the way to go, just know that it is an expensive process.
