When I took a job as a contract “systems engineer” for a large ship builder, I had no idea what I was going to be doing. Looking over my resume and seeing the (on paper) impressive track record I had for large software development projects – thank God resumes don’t typically list the current condition of the project! – the interviewer had warned me right up front that I wouldn’t be doing any coding. That was fine by me. I was a little gun-shy of coding at this point, having just seen a huge consulting firm go bankrupt (literally!) because the project I was lead developer on failed to deploy on schedule. So something different seemed like just the thing.
In the interviews, we talked about design documents, architecture documents, requirements gathering…all good, safe, non-coding tasks. Besides, it would be interesting to be in on the startup of a large project for a change. I was typically hired just as the smoke started appearing and right after the development team all quit and I was a little weary of salvage work. I was actually looking forward to a nice, clean start without chaos and disaster.
My first meeting was an experiment in chaos and disaster. Apparently there were five different development teams, and they were all short of developers. They could smell one of their own and they started circling me like a den of wolves circling a rabbit. They all seemed irate, stressed, and desperate for an experienced developer. No matter how many times it was pointed out that I was there to do Systems Engineering work, they continued to probe my availability to help with “the most critical bugs” that were holding up testing and causing them to miss deadlines.
This was confusing. How could they already have “critical bugs” before the project ever started? Those usually come in towards the end of development, not the start of it. And how could they be holding up testing? Why did they even have testers on staff at this point? What could they possibly be eager to test? And how could they be missing deadlines when they didn’t even have their architecture and design work done? Why did they even have coding deadlines?
Confused, I faded into the background and just observed. The more I heard and saw, the more I was reminded not of a project that is just starting up, but one that is just starting to fail. All of the signs were there. I had been duped.
Right after the meeting, I went to my new manager’s office for some clarification. It turns out that the project was not just starting, but had been going on for nearly two years on an 18 month contract. They were behind schedule, low on budget, low on developers, and working 18 hour days. They had hundreds of critical bugs and very few people to fix them. Now the development managers were out for blood because a Systems Engineering manager had sniped a perfectly good senior developer for what? Documentation? They were livid.
Then it all started making sense to me. I was indeed brought in to help create the architecture documents, the software designs, even the requirements. But at the END of the project, not the beginning. Apparently management was getting spooked by being behind schedule the persistent rumours floating around that the shipyard was considering cancelling the contract and refusing to pay for a failed project. This was going to cost the company many, many millions of dollars. Failure for this project to work could cause stock prices to fall, throw existing ship contracts behind schedule, and endanger future contract negotiations. Realizing this, and not wanting to be the ones the finger was pointing to after the fact, the Systems Engineering managers went into a frenzy to backfill all of the documentation that should have been done up front. That is what I was hired for. I was a contract CYA agent.
What was the point? Why would anyone be so worried about documenting software that was probably never going to be delivered or used? Why not just focus all efforts on getting it deployed and then worry about doing the documentation? It wouldn’t be the first project I had been on that had taken that approach.
It turns out that after a failed project, the company requires a “post mortem” audit. This is is basically a formalized witch hunt disguised as a “learning process” that brings in independent auditors who go through the tens of thousands of documents to try to determine why the project failed and how to avoid projects from failing in the future. The main problem with this in our case was that there really weren’t many documents to review. That was a serious contractual violation and would result in some heads rolling, so it needed to be “fixed” as quickly as possible.
What this all boiled down to was that I was going to be part of another crash-and-burn project derailment, but I was going to be creating the documentation that should have been created before development started. This is a very thorny job to have because it required us to schedule meetings with developers and have them explain to us how the software they wrote works so that we could document it and create a design for them to follow in creating the software. Very few of them saw the irony of it and even fewer were happy to work with us. They saw it as an effort to deflect blame to them and were less than cooperative. The development managers were even worse. They were downright hostile, and I can’t say I blame them. I had been is a similar situation on my previous project and I knew that asking a development manager to allow me to take up hours of his developers’ time when they were already way over deadlines was slightly more endearing than just kicking them in the groin.
The worst part was that, no matter what I did, there was nothing I could do to head off the coming project meltdown. I was just there to cover tracks. Or so it seemed.
Part 3: Duck and Cover
