Guardian Software

In order to show how mapping each item in a project to a CII an make managing the project easier, it is necessary to define some type of structure that we can talk about. In this instalment I will talk about the structure of an CII and why it is so flexible.

As I have already pointed out, the first thing we know about an CII is that it has a unique identifier. This can be in any form that you have a way of guaranteeing is unique. If you are a database-knowledgeable person, this would be equivalent to a unique key in a table. It must be entirely unique within a project. For the sake of simplicity, we will use an incrementing number for our examples, although in a real life situation I would opt for a Universally Unique Identifier (UUID) to guarantee uniqueness across projects in different databases. In addition to a unique identifier, what else do we need to manage an CII? If you remember, we also talked about the need in IM to version everything. That implies that an CII should have a version associated with it. It should. Each time the item that the CII represents changes, the version should be incremented.

So far we have the following:

ID (unique identifier)
Version ID (incremented)

This is a good start, but that can’t be all. I said that each CII represents an item in the project. How do we know what that item is? Is it a .dll file? A configuration file? A SQL script? A user document? It seems that a good thing to know is what we are dealing with. Maybe I don’t want to see all ICIs, only the ones that are user documents. Storing an indicator of what the underlying item is would be helpful in determining which ICIs I wanted to work with, so we should add a ItemType to our CII.

So far, so good. We can uniquely identify this item, we can find out what version it is at, and we can get a general idea of what it is. But what if I want to actually get a list of the documents, not just a bunch of unique IDs? In this case, we would want to add a Source File value to it. But wait…what if I want to only see the user documents that are for the system administrator? Should I add an “intended audience” value also? That seems to make sense unless you consider that we could be talking about a .dll file instead of a document. Now “intended audience” makes no sense at all. We seem to be headed down a dead-end road.

If you take this discussion to its logical end, you will realize that everything that could possibly be tracked about ANY item in your project will need some place to put data. This little CII is quickly starting to turn into a massive data structure with potentially hundreds of data items. It is a database designer’s nightmare! What to do?

If you think back to the last discussion, I talked a little about the powers of abstraction. By ignoring some things, or looking at them at a higher level, you can do some pretty amazing tricks that are not immediately obvious at the lower level. In some cases they are downright impossible. But if we re-frame the problem, a solution will emerge soon enough. Lets pretend for a moment that we are not talking about a project, but a retail order. An order can have a single item, or several items. Come to think of it, an order can have an unlimited number of items. How it this normally handled?

Skipping over the technical explanation, the answer is that at the order level, all items, no purchase what they are, are handled as “order items”. Like ICIs, they can represent anything in inventory. With this approach, we don’t have to worry about whether or not an individual item has a particular value because it is still an order item and can be treated like all others. While they may have some common pieces of information (stock number, quantity, price, etc.), the products that these pieces of information represent are really not very much alike.

Coming back to ICIs, we identified that there are several things that are common between all ICIs, but potentially many more that are not. The way to handle this seemingly impossible situation is to abstract this never-ending pile of possible data items down to a single thing: a Property.

Let’s take a look at what that does for us. For our purposes, a property is defined as “a piece of information relevant to an CII”. It is obvious that we must be able to associate multiple properties to a single CII, but once we come to that conclusion, the vexing problem that we were facing melts away. Now a .dll file can have certain properties (file name, size, install folder, etc.) while a web server can have other properties (Manufacturer, model, processor(s), RAM, etc.). By breaking apart properties from the ICIs that they are associated with, we have solved the problem of how to anticipate all possible pieces of information to be tracked.

The solution is that new properties are created as they are needed. This solution, however, leaves a problem to be solved. On one side of the room we have ICIs and on the other side of the room we have a big pile of properties. How do we know which properties go with which items? This is a fairly easy problem to solve logically. With retail orders, each item has an associated “order number”. The order number is unique per order (just as an CII id is unique per CII). When an item is added to an order, a line item record is created that contains both the order number and the product code for the item ordered as well as other pieces of information. Now we have a link that ties our item ordered to a particular order. To reconstruct the order, all we have to do is retrieve all line items that match the order number.

This is similar to what is done with an CII. As properties are added for an CII, they store not only the property data, but also the unique CII ID.

A simple puzzle. You have a room full of kids and no information about who they are. You don’t even know their names. You also have a list of parent names. How do you figure out who each child’s parent is? The answer is pretty obvious; you ask the kids. The point is, the kids know who their parents are so you don’t need any other method of figuring it out. in fact, you don’t even need to know anything about the child, you just need to ask what their mom or dad’s name is and find them on the list.

Similarly, in the CII scheme we are discussing it is very easy to figure out which property goes with which CII because the property “knows” who their parent is just as the order items “know” which order they belong to due to the order number “link” that each one contains. This is referred to as a “linked list” in programming and is a common method for linking a single item to multiple sub-items.

Armed with our new technique of defining properties as they are needed and linking them to an CII, we can now put the problem aside of figuring out what an CII looks like. It is sufficient to say that an CII has a unique ID and one or more linked properties. Problem solved! We have come full circle and in the end, decided that the only thing we need in an CII is a unique identifier value. Everything else can be linked to it as we need to add information. We will leave the actual “how” of linking properties to ICIs for a later discussion. Just trust me that it can be and is done every day in software that you probably use.

There seems to be a axiom emerging as we continue to discuss the methods we can use to create simplicity from complexity and order from chaos. You will see this theme emerging time and time again as we progress because it is a very simple and powerful technique. What it boils down to is “If you don’t know the answer, try changing the question.” Not something you are likely to get away with in other parts of life unless you are a politician, lawyer, or philosopher, but it works in Integrity Management quite well.

Now that we are mentally at peace with our ability to forge ahead without having to know and account for everything we may encounter along the way we can start looking at ways we can leverage this new-found power to make the job of creating software more reliable and easier.

Last time I talked about how every part of a project must be uniquely identifiable and explained how the Integrity Management Item (CII) can be used to perform this task. What I didn’t really explain was what an CII looks like, where it lives, and what it can do other than be unique.

While being unique is a full time job for many celebrities, it is not extremely useful on its own. If we are going to go through the trouble of giving something a name, it should do a little more than sitting around being unique to earn it! The CII does. It should provide some practical and useful information of functionality.

First, let’s talk about the structure of an CII. The question of “what does an CII look like?” is comparable to the age-old mystery of “how long is a piece of string?” In the case of the string, it is as long as it is, and in the case of the CII, it looks like it looks. So much for practical, useful information! The reason for giving this apparently useless description (other than wanting to use the string question) is to drive home the point that the CII is a concept, not a thing. Like a color, it can be useful but it is hard to pick one up and put it in your pocket (no, that is a crayon you are thinking of…only school kids call them “colors”!).

This is because as useful as colors are, they are not things. We can speak of them (I have on my blue shirt today) and use them to convey information (the blue lights in my rear view mirror); colors just cannot exist on their own. Instead, they describe other things that already exist. We can even group things logically by color that have no other common properties (make a list of the blue items you see in the room). In many ways, this is what IMIs do. They provide a convenient way to talk about and manage things that may not be very alike otherwise.

In the software world, as many other disciplines, we say that the CII functions as an abstraction of the various pieces that make up a project. It gives us a way to perform the same actions on very different things that would otherwise be difficult or impossible. For example, your project may contain a configuration file and a packaged database system.

Other than being file-based, these items have very little in common. If they were people they would stand on opposite sides of the room at cocktail parties. But what if the hypothetical people are both huge Dolphins fans? This is a common thing that they have in common that they can both relate to, so while the conversation is on politics, they don’t talk. Mr. DBMS is a Republican and Mr. Configuration File is a Democrat. When the conversation turns to movies, still they have no common ground. DBMS is an action movie fan while CF only likes documentaries. But when the subject turns to football, suddenly they are both interested and you can’t get them to quit talking about football. That is until DBMS’s wife, Web Server, whispers that he should talk to some other people for a while and stop boring everyone with football stories…then they again drift off to opposite sides of the room.

Pretty dramatic, huh?

The purpose of an CII is to provide that common ground between all of the different pieces of your project. Like Army soldiers’ proclamation of “I’m not black or white, I am green” and the even broader “soldier fist” motto, IMIs ignore the differences between the pieces and leverage the commonalities. No matter what a soldier’s job is whether field cook, generator repairman, or helicopter pilot, you can assume several things about him without even knowing him. First, he knows how to fire and field strip an M-16. He knows how to throw a grenade without killing himself, and he will put his life on the line to defend his flag. These are things that are common to everyone who wears the Army uniform. They may have different taste in music, different jobs, different religions, and different political views, but when an officer barks out “attention!” they will all snap to attention in the exact same manner.

In the same way the military trains each of its members with common skills and responsibilities, IMIs define things about any item in a project that can apply to every single one, no matter what it is.

This may sound like some kind of shell game of pretending that things are the same when they aren’t, but it isn’t at all. The key to understanding how to implement IMIs is to understand that nobody is born with the reflex to snap to attention when an officer yells “Atten-hun!” (Anyone who thinks they yell “ten-hut” has never actually been in the Army). This is an artifIMIal behavior that is introduced in their training to allow leaders to control them as a single unit when needed. Using the techniques collectively know as “drill”, a leader can move 1000 soldiers with the same effort it takes to move 10. This is significant because anyone who has ever been in a crowd of 1000 or more people trying to move from one place to another will have the word “mob” in their mind. With so many different people with so many different ideas about how to get from point A to point B, the resulting chaos is typically extremely ineffIMIent. By introducing some artifIMIal behaviors into this crowd, they can be treated as a single unit, moving in step and in perfectly straight columns, turning, stopping and even saluting in perfect order. This is the power of abstracted behaviors.

It is important to note that introducing these standardized behaviors does not in any way diminish the individual’s ability to do their job or move about on their own when coordinated movement is not required. The same soldier who marches in perfect synchronization with 999 of his fellow troops can, in the next moment, take up a lone position and effectively protect it without any further orders from his commander. He can even be left at this post for days and will perform his job as required with no further intervention. But when he rejoins his platoon and the “FALL IN!” order is given, he will be just as effIMIent in moving with the others as he was before.

Those who have partIMIpated in marching bands will also recognize the parallels between IMIs and marching drill as well. Each member has their own part to play (literally), but it takes a common set of drill steps and maneuvers to control the overall musical production, and marching band drill is based on the same concepts that military drills is.

The purpose of an CII then is, on an abstract level, the same as the purpose of marching drill. It defines a set of characteristics, rules, or behaviours that can be relied on no matter what the underlying item it represents is. For those familiar with object oriented programming, an CII could be considered a base class that other objects inherit. They all have the same basic structure but represent different distinct items.

Once we have established what the structure of our CII is, we can then “map” it to any item in our project whether it is hardware, software, documentation, or even other abstract things such as routine maintenance tasks and procedures.

Now that you understand the concept of what an CII is and how it is used, next time I will talk about how bringing all project under the collective control of IMIs can impressively simplify some of the most complex tasks that face projects as they move through the development, testing and deployment phases. Hope you will join me.

-M@

In my last article I talked about the need for a unique identifier for each Integrity Control Item (ICI). I am going to expand on that concept this time around. If you end up following this series all the way through, when you look back you will see that the IM is so critical to getting control of your project that without it, there isn’t much chance of pulling it off successfully. It will, one day, seem obvious.

The Integrity Control Item is a very basic unit. It is used to represent the smallest practical unit in an Integrity Controlled project. What is meant by “smallest practical unit”? The way I like to think of it is to compare pieces to atoms. That probably sounds a little off the wall, but having a high interest in Physics, that comparison just makes the most sense. Let me explain.

Everything in the universe is made of something. Well, in theory anyway. I have a personal theory that once physicists followed the route of existence down from “rock” to “mineral” to “element” to “molecule” to “atom” to “electrons, protons and neutrons” to “quarks” to “sub quarks and gluons” we are in grave danger. I can just imagine the fateful day that some physicist in a high energy lab somewhere makes the discovery that the entire universe and everything in it is made up of absolutely nothing. Suddenly everything disappears, leaving the limitless vacuum that everyone seems to agree once existed. Sounds like a Douglas Adams story.

Like physical matter, everything in a project, no matter how complex that project is, is made up of something. The closer to that something you get, the less the things differ. A ship is a very complex system, but if you divide it into hull, deck and superstructure, it can be logically separated. Designating decks, compartments, on down to individual pieces of equipment give an even more detailed view. Now we are standing on the deck of a ship looking at a single bolt. Is that the smallest practical part of a ship? I would say it is one of many. In fact, if you think about what we just did, something interesting is going on. Call it Conservation of Complexity.

We started out imagining a ship. It is a single item. If you saw it coming into port, you would not tell someone “wow, look at that massive collection of steel plates, bolts, pipes, wires and paint!” You would more likely say “wow, look at that huge ship! How in the world does that thing float anyway?” But that is a different discussion. The point is that the ship you see pulling into port represents a single item, but an extremely complex one.

Since the human mind is only capable of focusing on one or two items at a time, we see a ship, not all of the individual parts. But something weird happens. Once you start looking at it as a collection of parts, the overall number of items skyrockets. Instead of a ship, you have a hull, deck, and superstructure. And within that superstructure you have windows, doorways, passages, rooms, radar dishes, radio antennas, etc. If you were to examine a single room, you would find innumerable parts make it: steel plates, beams, insulation, pipes, wires switches, valves, and fixtures. If you were to examine the sink in a room you would find that it contains a myriad of parts as well: brackets, pipes, valves, screws, gaskets, etc. Examining the drain pipe would reveal individual sections of curved pipe to make up the trap, nuts to hold them together, gaskets to prevent leaks, a ball valve to close the drain, linkage to the drain stopper lever, and on and on.

But have you noticed that although the number of things in the ship has grown exponentially by examining them at close range, their complexity has decreased. A single room is much less complex than a complete deck. A sink is simpler than a room, and a drain is simpler than the whole sink.

Yes, that is obvious, so what? Why do I care?

Because ships offer a very good model for software projects. Like software projects, they are extremely complex, they require many people with many different skills working together in a very organized manner. Everything must be completed in the right order to the right specifications in order for it to slide into the water after having a bottle of bubbly broken on its bow. The effort required to plan the designs, procure the materials, control the order that things are built, make sure that steam lines are not run next to fuel tanks, and the millions of other details that go into building a ship is at least as great as the most complex software project. Yet ships do, generally get built and most of them don’t just sink when they are launched. This fact offers hope to those of us who feel that software is just too complex to do correctly. So are ships, but they get built.

So how do shipbuilders handle the overwhelming complexity of their trade? In short, they choose simplicity over component size. They design things from the top down, but they keep working it until they get down to a “lowest unit” level. They may be building a hull, but they do it in small job-sized pieces.

If a group of engineers got together one Monday morning to start developing a new ship design and for whatever reason decided that they would start with the sinks in the living compartments and work up to the overall design, how long do you think that would take? How correct do you think the designs would be? I think they would be very wrong indeed. Unfortunately since the design is from the lowest level to the highest, they would not discover that they had neglected to consider where the water for the sink was going to come from, or possibly the drain emptied into the fresh water tank below. In order to fix these problems, they would have to tear out huge sections of their work and constantly rework it. Sounding familiar?

Not that these things don’t happen. My brother works for a major US shipbuilder as a welder. He told me an amazing story one day that illustrates that sometimes even shipbuilders get it wrong. He was working diligently to finish up a welding project on a wall insider the belly of a new ship. He noticed as he was working that another person was setting up a cutting torch and then just waiting. “Am I in your way?” my brother finally asked him. “Nope, go ahead.’ was the answer. Half an hour later the same guy was still standing there waiting. Finally it began to bug my brother and he asked again, “Are you sure I’m not in your way?”. “Nope, I can’t start until you finish and get your work signed off” was the reply. “Oh, OK. What are you going to be doing?” Imagine the dismay when my brother heard his answer “Cut out that wall you are welding. We have a design change and that wall is moving.” This may seem outrageous…OK, it IS outrageous, but if you really think about it, you have probably seen some comparable things happen in software projects.

Circling back around to Integrity Control Items, software projects are very complex, but one way of getting a handle on that complexity is to take a tip from shipbuilders and track each part individually. This is what IIs do for us in Integrity Management. They define a single unit (for example, a .dll file or a file server) and allow it to be treated, for the most part, exactly like any other unit. This greatly simplifies the day to day things that have to be tracked, but it also creates a large amount of data to be tracked. So, as was hinted at earlier, we have traded complex large items for simple, numerous small items. Like the cans of corn and bottle of salad dressing with UPC codes mentioned in an earlier article, we can now operate on these very different items with the exact same logic.

Hopefully this has gotten you thinking a little about how complexity can be conquered by decomposing the complex items into many simple items. Next time I will dive a little further into how this is actually done in the real world.

-M@

Some of you older folks may remember a time when grocery stores put price stickers on everything. Each can of green beans or salad dressing had a white or orange sticker on it with a price. No barcode, just a simple number like “$1.29″. When you checked out, there was a cashier who could (brace yourself) actually type numbers into a cash register. In fact, she could typically type them very fast with one hand and move the groceries to the bagging area with the other.

One day I noticed that some things that I bought at the grocery store had a new “built in” tag with a lot of lines on it. It seemed completely useless to me because the only thing on it besides the row of lines was a seemingly random series of numbers. I eventually learned that these were called “UPC” or Universal Product Code tags. Since the tags were implemented long before most grocery stores received and installed the cash registers that could read these tags, they seemed rather pointless to most people.

But soon all of that changed. Because each product carried a unique code, and that code was the same for that product whether it was on a store shelf in Akron Ohio or Pascagoula, Mississippi, products could be paid for, inventoried, and ordered all using the UPC number. In fact, those processes eventually became completely automated. Modern checkouts at large chain stores such as Wal-Mart or Target read the UPC code, look up the price, subtract a single item of the matching type from inventory, and if the supply is getting low, automatically order new stock. There are currently some experimental stores that constantly monitor supply and demand and adjust the price of each item in the store based on these real-time statistics!

It is difficult to overstate the far reaching impact the simple UPC has had on the retail industry. It has enabled stores with millions of items in stock to instantly know how many of each item is on the shelf, how many are on order, and how many were sold in the past hour.

That is all very interesting, but what has it go to do with what we are talking about? I am about to tell you!

Before the days of UPC, the task of pricing, inventorying, and ordering stock was very time consuming. Since the cashier only had the price that was paid for an item to enter (not what the item was), there was no way to know how much the inventory of a particular item had been depleted other than manually counting items or, more commonly, “eyeball ordering”. A department manager ordered new stock when the stock on the shelf looked low.

In many ways, the problems they faced had a lot in common with the problems we now face on software projects. We have a huge number of individual pieces which must be assembled in just the right way to work. To make things worse, they are always changing. It would be as if there were gremlins in the store that constantly rearranged the shelves when nobody was looking and mixed different items up. Actually, I think those really did exist. They were called kids. I know because I mixed up a few shelves myself. With this overwhelming amount of information to track and with many, many items changing all the time, something is needed to get the kind of control on project pieces in a similar way UPC codes allowed stores to get control of inventory. We need a UPC for our project pieces.

Enter the Integrity Control Item. This is a concept that treats anything that is versioned (i.e. everything we care about) as a unique item. Similar to the way a can of Green Giant whole kernel corn has a code that is unique throughout the food supply system, our ICI is unique throughout our project. Actually, properly implemented it is globally unique even between projects, but more on that later. For now it is sufficient to say that a ICI is unique in a given project. In our examples, I will just use a numeric value like “324398675″ to represent an ICI.

So what does that do for us? Great. Now instead of worrying about the Oracle Client drivers, I can worry about keeping up ICI #324398675 and its thirty eight thousand brothers. Yea, buddy…that sure made my life easier! But wait…there’s more!

If the only thing we got out of an ICI was a hard to remember number it would be no better than a cell phone, but that is not the case. When a grocery store orders something with a UPC code, when the order comes in, it has additional information with the UPC code.

For example, it may have the manufacturer, the product name, the quantity, and the country of origin. You don’t need to know the country of origin to order the corn, you just need to know the UPC code, but if you need to know what that UPC represents, a large amount of information about it is available in the inventory control system. Similarly, if we want to automatically check that an ICI is present in a particular release, we don’t need to know what it is or what its characteristics are ahead of time, those can be looked up when they are needed or not looked up when they are not. There are many, many other advantages to tracking everything in your project with an ICI which will become more obvious as we dig deeper into this process, but for now I will let you ponder these ideas and try to shoot holes in them.

Next time I will discuss how versioning works with an ICI. I hope you come back to read about it.

-M@

What is “Project Integrity Management”? How is it different from Configuration Management? How does it make it any easier to prevent the dreaded project meltdown? I’m glad you asked.

As mentioned in a previous article, Configuration Management attempts to control elements of a project, specifically controlling and managing the constant change of information associated with the project. The amount of information to be tracked can be overwhelming, and the task soon becomes too complex to be performed manually. This leads most projects to seek a “toolset” to manage their configuration for them. While many commercial and some open source toolsets exist today, they all seem to be solving different problems. This is most likely due to the variations in different people’s understanding of what Configuration Management consists of.

What many projects call CM is actually not configuration management at all, but simply source code control. For this task, there are a number of software packages available that provide varying degrees of functionality and frustration. But source code control is not configuration management. I would even go so far as to say that it is not even part of the configuration management process, but I will save that argument for a later time when I feel like starting a flame war.

In the interest of avoiding splitting hairs on what Configuration Management is or is not, I have opted to create a new term altogether that will represent the concepts that I feel are the ones that need to be defined in order to get a handle on the mess that many of us know as the “development project”. This term is Project Integrity Management and although it is similar in some ways to Configuration Management, the two are not interchangeable, at least not with the definitions I have seen of CM. So from here on out we will be speaking of IM instead of CM unless I am referring specifically to a Configuration Management concept.

Definition: Project Integrity Management – The methods utilized to ensure that a project remains in a consistent known state throughout the product lifecycle.

First of all, IM is not source code control. This has been done many times over and, although I don’t think that the approaches that are currently being used are great, I have nothing to add to that debate. It isn’t that I don’t think source control is important, I just think that the process is well understood and we should focus on some of the more vexing problems that have not been so thoroughly studied and developed for.

At the highest level, IM consists of the two following areas:

Versioning
Environment Management

These two concepts represent everything needed to completely track and control a project’s variables. As you may have guessed, there are many lower levels to these concepts which we will explore, but first I would like to describe what is meant by these two terms.

Versioning is the most important concept in IM. Without versioning, the rest is impossible. In PIM, everything is versioned. If it is not versionable, we are not interested in it because the definition of “versionable” is “possible to change“. If something is impossible to change throughout the life of a project, then why would we care about it? It is in an eternally known state and tracking it would be silly. What would we track?

The catch in this definition is the very important word “possible” is a pretty wide-open one. For example, you may consider that the operating system that is being developed against is not going to change during the project, but you would likely be wrong. Service packs change operating systems all of the time. Operating systems even have their own versioning scheme already built in. Servers can crash and be replaced with different machines. Documents can be edited, corrected, and revised.

The upshot of this is that pretty much everything associated with a development project can change. Nothing is “set in stone”. What that means in IM is that everything must be versioned. Yes, that includes operating systems, servers, and documents. If it can change and it is part of a project (i.e. part of a deliverable), it must be versioned. We will explore the finer points of versioning soon, but for now this short discussion will serve our purposes.

The other high level concept in PIM is Environment Management. This may or may not be a new term to you; I didn’t make this one up! But I am going to heavily redefine it to suite PIM’s purposes. What I am referring when I mention Environment Management is the control of all versioned items in each project environment. Examples of environments in this context are “development”, “test”, “training”, and “production”. However, as you will find out, this is not to say that there is only one development or test environment. In most projects there are several. The PIM approach to Environment Management is one of “Just In Time” environment creation and destruction. Environments are like Krispy Kreme doughnuts. As soon as they are created, they start going stale, so they are best while still warm and soft. I will also provide a detailed approach to implementing this JIT environment control plan.

What this means is that the environments themselves are versioned. From a practical standpoint, this means that if product testers are testing a fix to an older version of the product, they can re-create the specific test environment that matches the software version they are testing, right down to the system configuration settings and hardware driver versions.

Those are the high level concepts. On their own they are not very useful though. In fact, at this point, they may not even be comprehendible to any meaningful degree. But if you could see the picture in my head (which I hope to bring you around to doing), you would see that not only is it comprehendible, but very elegant and flexible. Now that we have gotten the mile-high view, in my next article, I want to jump to the other end of the spectrum and give you a peek into how this can be pulled off. But first, a quick trip to the store.

M@

I would like to start by admitting that I am cursed.

SPOILER WARNING: if you are considering hiring me, please stop reading now!

I am an albatross to companies that hire me. It is unexplainable, but true. It is not that I sabotage them or fail to do my job, it is just that I have an uncanny ability to take jobs for companies that are starting to unravel. For whatever reasons, I have had a really bad run of projects in my career as a developer. It would be difficult to go into detail without naming particular companies that most of people in this industry would recognize or may have worked for, but suffice it to say that I have witnessed not only the crumbling and collapse of many large projects first-hand, but I have also been around when the pink slips started going out and servers and network equipment were boxed up and sold off and the office spaces put up for rent. Thorough all of these tragic projects I have been baffled by the inevitable decline, unravelling, then total collapse of projects under their own weight.

Certainly a lot of the problems I have witnessed have been poor management in the form of lack of market research, grossly underestimating required capital, reckless spending, questionable accounting practices, too heavy reliance on new technologies, and misreading of the bones in general. In some of the cases, the project was dragged down as part of a larger collapse of the entire company or even industry and it would be inaccurate to blame those failures on the project itself, although I am quite sure that if the companies’ problems hadn’t killed them, the projects themselves would have self-destructed further down the road. I know that some people reading this are puzzled by my eternal pessimistic view of software projects because they have had the extreme fortune of working on projects that were well planned, well managed, and held to the high standards that any complex project should be. To these people, I say “count your blessings!”

This particularly bad run of luck with projects has produced a feeling that the software development world is a wild west “free for all” and that even the largest companies seem to be incapable of managing a software project to the same level of professionalism that they would manage other large projects. This is what has led me to make the much criticized comparison of software engineering projects to other engineering projects. I have received a lot of comments that it is a brainless comparison because building software isn’t the same as building airplanes, cars, or bridges. There is some truth to those criticisms, but I fear the folks making them are missing my point. My point isn’t that they aren’t different, or course they are! My point is that we seem to be making little or no progress towards stabilization of software projects.

The main argument I have seen for why everyday software is so buggy is the one of costs and market forces. The argument goes “yes, you could have solid software. NASA has shown that. But the price of such software would be astronomical (pun intended) and nobody would be able to afford it.” This seems like a legitimate argument on the surface, but I think it is flawed for several reasons.

First, software is a weird creature when treated as a product. It is generally sold the same way computers or keyboards or cameras are…as a product. But in reality, it is more of a service. A group of developers has provided the service of codifying complex business or other types of processes into a set of rules that can be applied by anyone with general knowledge of the business and basic computer skills.

Don’t believe me? Then think about this: what are you buying when you spend $499 for a piece of software (or 499K for that matter)? Surely you aren’t paying that amount for the actual disc. Even if it has a printed manual (yes, they used to do that!), the actual value of a couple of CDs or a DVD and a book is somewhere below $10. Some will say that you are paying for the privilege of using software, not the media or the software itself, and they are closer to the truth. When you purchase software, you don’t own it any more than you own the songs that you download from iTunes. You are purchasing the right to use it. But why would anyone part with their hard earned money for the right to use something without owning it unless they got value from using it? They wouldn’t. It is a given that if someone purchases something, it has a value to them. In music that value is the pleasure given by hearing that “YMCA!” remix and with software that value is whatever it is that the software does, makes easier, or makes possible that otherwise wouldn’t be. You are paying for the service that the company that produces it provided up front in automating some difficult task.

I am going down this path to make a point. The common argument I see is that NASA produces reliable software and it costs millions of dollars. Who could afford that level of reliability other than an agency with deep tax pockets? The common computer user certainly couldn’t. But what this argument misses is that NASA is pretty much their only customer. By the same logic, if a car can only be made reliable by investing millions of dollars in research and development, then a company that builds just a single car (as in one physical car, not one model) would have to charge millions for it just to break even. By this argument which is commonly given to justify sloppy software, nobody but the most wealthy could afford a reliable car. But that is not the case because this is not what car manufacturers do. They invest millions in making a reliable design, that is true. But then they reproduce that design at a minute fraction of the original cost. That design, reproduced tens of thousands of times, is then shipped to dealers who can afford to charge $25K for that same car that took millions to design and produce and still make enough commission to take a week long vacation to the Bahamas. In other words, the cost of producing a reliable car CAN be compared to the cost of producing reliable software if both have a large customer base. It is a valid comparison.

Once this point is understood, it stands to reason that software should actually be even easier to produce both reliably and cheaply than cars or airplanes. After all, each car produced contains several thousand dollars worth of raw materials and requires an appreciable amount of labor to produce. In addition to that, there are shipping, storage, and depreciation costs to consider. Software, on the other hand, is an all up-front prospect. Once the original, fully operational version of it is produced, the cost of producing another copy is negligible. So is the cost of producing the next copy, and the next, and the next million. The only raw materials needed are a couple of CDs or, more commonly, bandwidth on a network for providing downloadable copies of the software.

In this way software is more like a book. The original takes the author many months to prepare, but once it is complete, little or no further work is needed to regenerate it. If it is an eBook, no effort at all is required. Another download is just another opportunity to make money from the original. The risk in this model is that the author will spend months of his life writing a book that nobody will pay for. If this turns out to be the case and the author has figured his time as worth a certain amount per hour, then it is all in vain and he has lost a lot of time and income. But if he suddenly got a buyer for the first copy of his book from an individual, he would have to charge tens of thousands of dollars to that one customer to just break even. We all know that isn’t the way it works though. A book provides value, but not that much value! The market will support a certain price level and that is what is charged for a book. If enough books are sold, the author makes money. If not, it is a wash and he takes some level of loss. This is a system that has worked for many years and has rewarded authors who provide relevant, quality work with a proportional reward and rewarded authors who produce poor quality work (in respect to value to customers) with no reward.

My assertion is that reliable software CAN be produced at a market sustainable price, similar to the example of writing a book. I know that I will get some arguments that somehow comparing creating software to writing books is invalid because (whatever), but that is the problem with comparisons. They are ALWAYS invalid at some level, otherwise they would just be restatements.

The problems I have seen seem to come in with custom development projects. They usually fall in the category of “one customer who must bear all of the cost”. These projects lack the economy of scale that mass produced products enjoy and are like NASA and FAA projects in this respect. One client must bear the full cost and the argument can be made that, faced with the enormous cost of producing extremely high quality software, most companies settle for “good enough” software for a much lower cost. There is a lot of truth in this argument and I believe it is valid. But the flaw here is the assumption that the only way to produce quality software is by dramatically increasing cost. The implication is that clients put up with sloppy work because they can’t afford high quality work. What this ignores is that there are many ways of controlling both cost and quality. In my next article, I will talk about some of the main causes of high cost and poor quality on software projects. See you then.

M@

In my last article I put for the theory that the most important thing to do when considering a new methodology, especially something as far reaching as CM (or as I have renamed it, Project Integrity Management, or PIM – I can do that, its my article), is make sure you have the support of your developers. They are smart folks and it is much better to have them on your side than against you.

So how do you do that? How do you get a bunch of “I’ll do it my way, thank you” hermits to agree to do extra work for the good of the project? The most obvious way to me is to show them that it really is for the good of the project. Most developers really do want their projects see the light of day and very few I have met actually want a project to fail. But you have to keep in mind that if they have been doing this kind of work very long they have been fed many lines about a new framework, a new development methodology, or a new tool that is supposed to make their job easier and the end result more reliable. They are going to be skeptical, and that is a good thing. I personally don’t want a bunch of gullible developers who jump on the next newest new thing that comes out. I like to see some caution when it comes to changing things that can make or break development project.

I once worked for a small company that prided itself as one of the only “Microsoft Premier Partner” shops in the area. They stuck Microsoft posters all over the place and hung the Premier Partner plaques (the clear plastic ones with Bill Gates “signature” on them) in the front lobby. They plastered their web site with Microsoft logos and “awards” (MVP, MCDS, MCP, etc.). It was pathetic.

Needless to say, when Microsoft came out with a new technology (or a repackaged existing one), our company president couldn’t contain his ecstacy. He would have the admins immediately load it on a server and start “taking it for a spin”. For the next two weeks we developers would get a constant barrage of emails with code snippets in them and subjects like “LDAP is so easy with this new _______” or “S-WEEET..Exchange interactivity via XML SOAP Protocol”, or whatever. You get the idea.

Inevitably within a few weeks he would have satisfied himself that this new technology was top notch and ready for prime time, even if Microsoft hadn’t (Or as he would yell from his office at regular intervals, “Rock On!”). He would then have all of the developers drop the work we were doing on real customer requested features and bugs and start recoding the entire app for whatever this new technology was. I was coding in .Net in late 2001 and early 2002, back when even Microsoft considered it Alpha. Not in “hello world” projects, but in the company’s main product, a very large home healthcare system.

To keep this environment going, you can’t afford to have any devlopers who were cautious or thought for themselves. They all had to toe the line and follow the Microsoft lead in everything. We added remote messaging to the application to report application errors. Why? Because these things called “sinks” in the Enterprise Application Architecture Library COULD. No other reason needed. He also had us require the user to enter a description of the error and what they were doing when it occurred. Apparently (as I predicted) there is a particularly troublesome area of the application commonly referred to as “asdf”, because that is what most of the replies contained. Some of them were more colorful and described what the computer (and by extension, we developers) could do with the error message. Little was left to the imagination on those. Did I mention that once we got the entire application converted to .Net (by then it was in Beta), we started converting the whole UI to Infragistics .Net Ultragrids? Those were beta as well and oh, boy what a bucket of fun that was!

Looking back, I see now that what that shop needed was a few skeptical developers who would stand up and say “that’s dumb and I am not going to do it”. At the time I was so happy to be back at work after a six month dry spell that I wasn’t going to do it. But I should have.

After this experience, I am naturally cautious of any developer that responds to an announcement that we are going to “re-architect”, “re-design”, or “re-implement” anything, much less the way we control the changes that happen in an application development cycle with anything other than outright objection.

So the first rule is:

If nobody objects, be careful; they are either new graduates who haven’t learned yet, or they don’t plan on doing it anyway. They are just trying to get you to go away so they can get back to coding.

If they are good at what they do, you will get instant pushback in the form of “Oh, so this is going to fix all of our problems just like “Agile” did? Or was it “CMMI” that was going to do that? No, wait, I think it was “RUP” or maybe it was ___________!

Don’t panic; this is a good sign because if they are skeptical and you bring them around, you will know that they are sincere about doing it, and if you have been led down the primrose path by me, they will let you know where the gaps are.

As I mentioned in my last article, the way to the developer’s heart by making his job easier, but only if it appeals to his built-in logic processor. Less “stupid _ _ _ _” and more development power is the key. Developers breathe logic. Their entire purpose for consuming oxygen is logic. A good developer can spot the holes in any logic from five miles up, so you have to know that what you are advocating is sound. Once you appeal to their sense of logic (“hey, you know, that makes a lot of sense!”) and you demonstrate how it will make their job easier by taking the bull off of them, you will win them over.

I know this because I am, at heart, a developer and I have never been convinced of the value of CM until recently. In fact, it took me approaching CM the same way I would any other project to get it broken down to the point that it made sense, which took me taking a contract job of designing a CM system for an existing project that was on shaky ground and knowing that I had better come up with something that made sense.

The next article will begin to describe a system of project integrity control that is based on very sound, very flexible and very simple logic. We will start at the most basic level of tracking and expand it to show how it can be applied to any project, no matter how complex it is. I hope you come back to read it.

M@