Configuration Management is probably one of the most dreaded and misunderstood tools that software projects have available to them. As a developer with over fifteen years of experience, I have to admit that I was blissfully unaware of it until four or five years ago. However, I had become very troubled by what I saw as the unexplainable way projects were crashing and burning around me. I found it difficult to understand how development projects with budgets with six zeros after them could just fall apart with no clear cause.
There were always reasons cited that sort-of explained why a project ran off the rails, but none of them were really convincing to me:
“Congress killed funding for the project”
Why? Could it be because the project was already 150% over budget. two years behind schedule, and becoming obsolete before it was ever released?
“The technology was just too complex to pull together”
Why was this a surprise? Couldn’t they have seen that before spending $20 million in development money on it?
“We ran out of budget and had to lay off a lot of key developers”
Why were you so far over budget to start with?
“The development team never did understand what we were really trying to do”
Why is that? How can a team work on a project for two years and not know what it is they are building?
The excuses for projects crumbling go on and on, and the most disturbing thing to me was that they never seemed to learn anything from their mistakes. Developers scattered to different companies that had the next great world-changing project starting up and eighteen months later were laid off looking for another gig.
In all honesty, as a developer it wasn’t so bad. I learned early on that the trick to being a happy “coding cowboy” was to never get too comfortable in your saddle. As long as you went into a project knowing that, chances are, this thing will explode on me before it ever sees the light of day, then everything else was pretty good. The pay was great, we were usually using extremely new and exciting technologies, and the atmosphere was almost always electric and exciting. While it lasted.
As I grew older and tired of moving to a new project every year or two, I began to question why projects never seemed to get it right. No matter how large or small, no matter what ex-football player or admiral was on the board of directors, no matter how much venture capital (or, as we called it in the development world, “vulture capital”), eventually the walls started crumbling and we shifted into “death march” mode where we were putting in eighteen hour days and consuming truckloads of Mountain Dew and pizza in a doomed effort to fix the mushrooming bugs and new requirements on an ever-dwindling budget.
These types of projects have been documented before. A book was written about one called “Dreaming in Code” which chronicled the rise and subsequent crash and burn of an open source project called “Chandler“.
The thing that I kept coming back to when considering all of these different project failures (other than “I must be an albatross!”) was that there was some hidden, ticking time bomb in each of them that at some certain point in the project exploded and blanketed everything with chaos. Suddenly, just as budgets were getting strained, deadlines had been missed, and people started getting burned out, everything seemed to just fall apart. Bugs appeared by the dozens, then hundreds, and then thousands, as if they were breeding. Usually about this time (when the first development releases were being tested and put in front of the users), missed requirements were discovered and had to be “grafted” back into the application. This, of course, resulted in more bugs that had to be fixed, and fixing bugs often led to new bugs, and on and on it went until the company ran out of money and dealt with it by laying off their most expensive people first, which also deprived them of their most experienced developers. Soon thereafter, the company would declare the project officially dead and move on to their next world-changing disaster.
What was this mysterious force that seemed to attack a project like a virus and bring it to its knees? How could so many smart people screw up so badly so many times? This problem really bugged me for several years.
In 2008, I was once again looking for work after the last company I worked for, a huge consulting firm that would have been kicked off the New York Stock Exchange for accounting irregularities had it not been kicked off sooner for trading below $1 per share for a solid month, went belly-up. The project that crashed and burned on me in that instance was an internal accounting system that was supposed to correct the accounting irregularities, but it failed to make it out of the gate due to the same old bugs and delays. In this case, a failed project actually did bring down a company of over 15,000 employees…and I was lead developer on the smoking project! This really brought the problem home to me. I had done everything I could conceivably do to make the project a success and thereby prevent the collapse of a huge company and had utterly failed. It weighed very heavily on my mind. I took some comfort in the fact that I had been taken off the project after insisting that we not completely re-engineer it from the ground up a fourth time, but at that point it was pretty obvious that it was doomed anyway. It was technically still on my watch.
While looking for another development job, and questioning my qualifications to be lead anything after the spectacular failure that got my development team and thousands of other people laid off, I was offered a temporary contract job at a large shipyard to do some “systems engineering” work. I had no idea what they wanted me to do, but I aced the interviews and they felt confident I was qualified. At the same time, I was wondering if maybe I should get out of the software development business anyway, given my track record of killing off otherwise viable companies. Maybe some “systems engineering” or hamburger flipping was just what I needed. So I took the job.
As it turned out, I went from the frying pan and into the fire.
Part 2: CYA Agent
