Why is software development schedule estimation so hard?

As seems to be becoming a bit of a habit with me, I've decided to post an answer (with a few very tiny edits) which I left for a question that I found on the Programmers.SE website, mostly because I feel it's a pretty good answer, but also because it was an interesting enough question that I felt it would be "blog-worthy".

So the question was basically about two things, which was effectively to ask:

    Why is it so hard to provide accurate estimates for a Software Development Schedule, and 

    why isn't this as simple as planning a construction process, such as building a bridge.

Software schedule estimation isn't really very hard at all once you have some experience and can base your best guesses on tasks that appear to be similar to something you have done before. So if (as I'm suggesting) it isn't hard to estimate project schedules, why does it seem so difficult?I'll start by answering the second part of the question first. Why isn't writing software more like bridge building? It's because of the level of input by each of the stake holders, and because of the perception by all involved at the ease with which software can change mid project. Change the design of a bridge, and you need to factor in significant material costs and the scheduling chaos that it can create organizing many individuals with a number of different skill sets. Software by comparison seems like it should be easier because all of the software developers are meant to be similarly skilled, and there isn't usually much in the way of materials or scheduling that can't be generally overcome (IE: delay is relatively cheap compared to delayed construction projects). The reality however is that software is all about dealing with people, and creating solutions to problems for people. When the project is so people-focused, it becomes less precise, and more affected by the problems that occur to and between people. Of course, with all of that said, anyone who thinks that a bridge building plan is easy to produce probably hasn't tried to build a bridge, and I am sure there are lots of problems that civil engineers face, just as we software developers do.

I believe that the difficulty comes down to several factors, and the most obvious ones that come to my mind at the moment are as follows:

• The developer doesn't fully understand the problem domain.
  
  You can blame the customer all you want for "not knowing their own mind", but the real underlying problem is that the customer is generally and genuinely not able to put into words what it is that they NEED. Wanting and needing are two very different things, and it is up to a skilled developer to draw out the customers needs, rather than merely responding to what the customer initially states as wants. When you are able to talk to the customer in their own language and have nailed what it is that the customer actually needs, then you are able to claim to fully understand the problem domain.
  
  
• Limited input from team members of varying levels of experience.
  
  Let's face facts. Nobody works well when driven by someone else's estimates. Every developer's experience is somewhat unique or at the very least different, and each will view a particular task from a different personal experience. If you only have one person estimating the entire project, they will only really get about 25-40% of their estimates correct, and it will be the difficult stuff that they have either no experience with, or which has been misunderstood that will end up causing the most delays. You can bet your auntie that the greatest delays will come from something a single estimator thought of as relatively trivial. This is where group estimation is very important. You get all of the people who will actually work on the project to provide all of the estimates (A planning poker session can really help here), and basically use this process to effectively hedge your bets against possible scheduling failures in the future.
  
  
• Developer hasn't tuned their Software Development Method
  
  There has been a lot of stuff written about software development methodology over the last 20 years. While I can't claim to have read everything, I can like most experienced people out there claim to have sampled a bit of everything, and have tried much of it. One of the reasons that I have become quite avid about being Agile is because of the emphasis it places on tuning your processes, and about being adaptable in both how you apply your practices, and in your thinking. Sometimes the method needs to be tuned to match the needs of the customer, but more often the greatest method tuning needs to be with the team itself. Whenever a new team member joins, it is an opportunity to tweak your processes, and when the team doesn't gel, you need to tune even further. Most serious failures happen because of teams that work ineffectively as a unit, and where the method doesn't (or can't) account for the unique situation of a given team. Everything from personal issues, to office politics, fatigue, experience, all of it can have an effect, but if the process doesn't make sense to the team, if it makes them feel uninspired or if it doesn't make the team feel in control or empowered by their efforts, then it's going to fail the team, and the team will end up failing the project. Processes that help team members feel like they are achieving good results in small manageable stages, and where team members feel rewarded by the effort itself, are processes that will ultimately deliver the best results to the customer.
  
  
• Plans are derailed by the most unexpected things
  
  As any military planner will tell you, you can have the best strategy all worked out with every minute detail planned, but once you start the battle the plan will be shot all to hell. Sure, you can load your estimates with likely delays, such as people getting sick, planning their holidays, or having that massive trade fair pop up right in the middle of your planned development period, but you can't always plan for things like massive incompetence, your entire server farm going down, your entire development team simultaneously contracting the Bengal Flu, or your suppliers suddenly going bankrupt. Things can, and most certainly will go wrong, and finger pointing won't help anybody. You can however plan to handle the unforeseen by negotiating alternative release schedules, and by ensuring that every release to the customer provides them with working software that delivers business value with every release. 

So by this definition, it's systemic failures that will derail projects, and not always specifically problems with scheduling. Scheduling itself is relatively easy compared to all the other stuff that a team needs to get right in order to deliver a successful outcome, and without all of that additional supporting "stuff" that is needed, it's unsurprising that many will find it difficult to provide decent estimates at all, let alone a schedule that they hope to complete successfully.