A project, typically because of contractual obligations, makes the formal declaration that they are going to follow some methodology that is supposed to make everything sweetness and light. They have various names (in order of “all the rage”): TQM, CMMI, ISO 9000, Six Sigma, ITIL and various others. Each of these “Management Methodologies” suffered from the same malady that the “Social Sciences” suffer from. They all try to explain everything, or in the case of management methodologies, they all tried to fix everything.

Of course the developers just wanted to code something, but with these processes, coding came way at the end of the line. In the meantime, an eternity was spent gathering requirements, creating goofy little charts with stick figures doing odd things, then sitting around in meetings talking about “actors”, “stories”, and various other touchy-feely things instead of cranking out the stupid code.

This lasted for a while, but slowly developers revolted. As a direct result of these over-burdensome methodologies, developers began rethinking things. Somehow they managed to get the idea that management should just leave them alone and let them write something, then let the customer tell them what sucked about it, then go back and fix those things, write some more things, and just keep doing this until they had an awesome software product. The main hints that it was developers, not managers that came up with this process is that meetings were discouraged, working with your buddy was required, and working flexible schedules and ignoring deadlines was mandated. If you weren’t going to hit a release deadline, so what? Just take that piece of code out and release what you have. They will hate it anyway and it will be back for rework in a few days, so why worry?

The name given to this developer’s erotic dream was “Agile”. Who could hate something that was agile? It sure sounded better in a sales pitch. Consider the difference:

“Our developers are all Six Sigma and CMMI certified”

vs.

“Our developers are all Agile certified”

If I were a clueless customer, my money would be on Agile every time.

The problem with ALL of these methodologies, Agile included, was that they never really addressed the core problem. I am not surprised by this because I never even realized what the core problem was until after I had already solved it. The other problem with them is that they were so complex, so time consuming, and so utterly head-banging dull that they were soon forgotten once development kicked off. Then towards the end of a project, managers would realize that there was no documentation to prove that they had followed said methodology (because they hadn’t), so they would pull a bunch of unlucky interns or contractors in and have them start creating history in the form of back-dated documents. Trust me, it is very common.

As I continued to work on the required Configuration Management Plan, some patterns started to fall into place. I realized that the reason projects go so bogged down and suddenly crumbled was often because they just got too complex to manage in the traditional ways of software development. At first, everything seems good. There is a set of requirements (we are assuming they are pretty good) and a team or two of developers. There is no source code. Everything that the project can and will be lies in the future, and that future is exciting. Coding starts. The first milestones are met with ease and the first release it sent to testers.

As testers find bugs (and they will find bugs), bug tickets are opened and worked by developers. Now there are two streams of work. Developers are still working on new code, but they are also working on fixing bugs in the existing code. Things get a little more complicated as new work is managed along side old work. If a project is using source control, they will likely branch the code at each release to test. Bugs must be fixed in the branch that they were discovered in, and then those changes must be propagated to the new branch(es) as well. A few more cycles of this process and things are really getting weird. Testers are not sure what environment they should be testing against. Most projects just resort to testing all code in the most current environment, but that environment slowly gets polluted and starts to smell bad. Phantom bugs are discovered that cannot be reproduced in the development environment.

Meanwhile, documentation efforts are ongoing. When a bug is discovered and the design must be changed to correct it, the existing design documents become incorrect. When missed requirements are realized, the designs again change and the design documents become even more incorrect.

Databases soon start to suffer the same problems as other components as their table structures, procedures, views, indexes and constraints are changed to keep up with bug fixes and new requirements. Some tables fall to disuse and some stored procedures become obsolete. Procedures suddenly expect parameters that the application code is not providing and data being returned to the application is not in the proper format or of the proper data type, resulting in more bugs.

What is happening here is that the project is getting complicated. There are many streams of work effort at various stages of completion, there are multiple environments that must support the changing code, and it becomes difficult if not impossible to manage it all.

The worst part of this often repeated scenario is that it usually gets worse the longer a project goes on. While budgets are being overrun, delivery dates are being missed, and developers are growing tired of sprinting, the complexity just keeps growing.

Complexity is not pretty.

The reason is simple to state, but difficult to correct:

Complexity  – Order = Chaos

At the stage that has been described, most projects I have seen are, whether they admit it or not, in total chaos. There is no logical way to predict what a change will affect, there is no way to estimate how long anything will take, and there is no way to be sure it is going to work in a production environment. This is the point where many a project manager will, despite what he has learned in management classes, just throw more people at the problem. As we all know, this just adds to the complexity as new developers come up to speed on the existing codebase and environments. Since what little documentation is available is almost certainly outdated if not absolutely wrong, the over-worked developers must take time out of their work day to teach the new developers what they need to know.

Chaos – Direction = Apathy

Without any real hope of getting the project back to a known good state, many developers become disengaged and apathetic to the deadlines and mounting bug counts.  Many start looking for a better project, sometimes from their work computers during business hours.

Apathy – Solutions = Death

Before long developers who have been on the project from the beginning are leaving. They hate the project and everything that reminds them of it. The dreams they had for it are gone and all that is left is disgust at the stinking, bug-infested mess that it has become. As the experienced developers leave, they often poison the well, advising the newer developers to get out while they can. Project death is soon to follow.

I have been through this cycle more times than I care to count. I never could figure out when it started, but I could sense it like something creeping in the dark behind me. I knew when death was near, and, coward that I am, I would be one of the developers looking for a new project (although I never did it on the job!). All of those times of going through this, I never understood why it happened, but as I was working through the details of this Configuration Management Plan, I began to slowly see what had happened, and I realized that it happened much earlier in the project than I would have believed.

In fact, a project’s death begins before the first line of code is written.

Just when I was seriously considering calling back one of the other companies who had recently made me a job offer, I was saved.

I had sat through several contentious meetings where senior management had directed the development managers to cooperate with systems engineers “without exception”, only to have the development managers loudly protest that it was all a sham and it wouldn’t do a bit of good to document software that wasn’t going to get finished if we didn’t stop wasting their time. These meetings often morphed into general gripe sessions where everyone was accusing everyone else of messing up their code, trashing their databases, or breaking their test environments.

Slowly a picture emerged in my mind of what they were trying to do. They had one test environment for all teams to use for general testing purposes called “sandbox”. This was the environment that was causing so much hate and discontent. With ten to twenty testers constantly testing what was often buggy software fixes, the data got more and more corrupted, versions of software got mixed together, and settings got changed, breaking other tests. It became apparent that the environment should have been named “Swimming Pool” because they were all peeing in it and complaining of the smell.

In a casual meeting in my manager’s office, I mentioned that I had dealt with this sort of problem before and may be able to offer some suggestions to ease the problem. As I was explaining, she began getting an excited look on her face (yes, I am charming, but not THAT charming). Finally she broke in and excitedly explained that one of the major documents that still needed to be finished was the Configuration Management Plan. I tried to hide a yawn.

As it turns out, she had been saying some of the very same things that I was saying about how to manage the environments better, but nobody was listening. She realized that the only way to get them to put aside their finger pointing and get them to actually do something about the problem was to trick them into agreeing to it in a procedural document. A procedural document such as a Configuration Management Plan. Of course she left out the tricking part and just explained to me that if I could come up with some procedures that would help manage the environments and document them in a way that was easy to understand, she would work on getting them to agree to follow it.

I saw an immediate benefit to this idea. It got me out of the meetings with people screaming at each other for hours about things that they had no idea how to fix. I took the extra step of letting her know that, in order to do that, I would need a quiet, out of the way place to work where I wouldn’t be constantly barraged by angry managers. She moved me to a desk in an adjoining building and stopped inviting me to most of the meetings. I started to see a light at the end of the tunnel. I was just hoping it wasn’t an oncoming train.

Knowing absolutely nothing about how to create a Configuration Management Plan, I started downloading examples from the internet and studying them. The first thing I noticed was that there seemed to be an extremely wide opinion of just what configuration management is. Some seemed to think it was a series of meetings while others seemed to think it was just another name for source code control. I can’t count how many times I saw arguments over which was better for configuration management: SVN or Visual Source Safe (hint, neither one are any good for it. They are for source code control).

I knew I was in trouble even as I was reading and walking back from the printer. Right there, in crisp black and white on the still warm paper were the tell-tale words: “The tool that shall be used for configuration management on this project is Visual Source Safe.”

I was in for an uphill battle.

Part 4 - Seek Not and Ye Shall Find

In the interviews, we talked about design documents, architecture documents, requirements gathering…all good, safe, non-coding tasks. Besides, it would be interesting to be in on the startup of a large project for a change. I was typically hired just as the smoke started appearing and right after the development team all quit and I was a little weary of salvage work. I was actually looking forward to a nice, clean start without chaos and disaster.

My first meeting was an experiment in chaos and disaster. Apparently there were five different development teams, and they were all short of developers. They could smell one of their own and they started circling me like a den of wolves circling a rabbit. They all seemed irate, stressed, and desperate for an experienced developer. No matter how many times it was pointed out that I was there to do Systems Engineering work, they continued to probe my availability to help with “the most critical bugs” that were holding up testing and causing them to miss deadlines.

This was confusing. How could they already have “critical bugs” before the project ever started? Those usually come in towards the end of development, not the start of it. And how could they be holding up testing? Why did they even have testers on staff at this point? What could they possibly be eager to test? And how could they be missing deadlines when they didn’t even have their architecture and design work done? Why did they even have coding deadlines?

Confused, I faded into the background and just observed. The more I heard and saw, the more I was reminded not of a project that is just starting up, but one that is just starting to fail. All of the signs were there. I had been duped.

Right after the meeting, I went to my new manager’s office for some clarification. It turns out that the project was not just starting, but had been going on for nearly two years on an 18 month contract. They were behind schedule, low on budget, low on developers, and working 18 hour days. They had hundreds of critical bugs and very few people to fix them. Now the development managers were out for blood because a Systems Engineering manager had sniped a perfectly good senior developer for what? Documentation? They were livid.

Then it all started making sense to me. I was indeed brought in to help create the architecture documents, the software designs, even the requirements. But at the END of the project, not the beginning. Apparently management was getting spooked by being behind schedule  the persistent rumours floating around that the shipyard was considering cancelling the contract and refusing to pay for a failed project. This was going to cost the company many, many millions of dollars. Failure for this project to work could cause stock prices to fall, throw existing ship contracts behind schedule, and endanger future contract negotiations. Realizing this, and not wanting to be the ones the finger was pointing to after the fact, the Systems Engineering managers went into a frenzy to backfill all of the documentation that should have been done up front. That is what I was hired for. I was a contract CYA agent.

What was the point? Why would anyone be so worried about documenting software that was probably never going to be delivered or used? Why not just focus all efforts on getting it deployed and then worry about doing the documentation? It wouldn’t be the first project I had been on that had taken that approach.

It turns out that after a failed project, the company requires a “post mortem” audit. This is is basically a formalized witch hunt disguised as a “learning process” that brings in independent auditors who go through the tens of thousands of documents to try to determine why the project failed and how to avoid projects from failing in the future. The main problem with this in our case was that there really weren’t many documents to review. That was a serious contractual violation and would result in some heads rolling, so it needed to be “fixed” as quickly as possible.

What this all boiled down to was that I was going to be part of another crash-and-burn project derailment, but I was going to be creating the documentation that should have been created before development started. This is a very thorny job to have because it required us to schedule meetings with developers and have them explain to us how the software they wrote works so that we could document it and create a design for them to follow in creating the software. Very few of them saw the irony of it and even fewer were happy to work with us. They saw it as an effort to deflect blame to them and were less than cooperative. The development managers were even worse. They were downright hostile, and I can’t say I blame them. I had been is a similar situation on my previous project and I knew that asking a development manager to allow me to take up hours of his developers’ time when they were already way over deadlines was slightly more endearing than just kicking them in the groin.

The worst part was that, no matter what I did, there was nothing I could do to head off the coming project meltdown. I was just there to cover tracks. Or so it seemed.

Part 3: Duck and Cover

There were always reasons cited that sort-of explained why a project ran off the rails, but none of them were really convincing to me:

“Congress killed funding for the project”

Why? Could it be because the project was already 150% over budget. two years behind schedule, and becoming obsolete before it was ever released?

“The technology was just too complex to pull together”

Why was this a surprise? Couldn’t they have seen that before spending $20 million in development money on it?

“We ran out of budget and had to lay off a lot of key developers”

Why were you so far over budget to start with?

“The development team never did understand what we were really trying to do”

Why is that? How can a team work on a project for two years and not know what it is they are building?

The excuses for projects crumbling go on and on, and the most disturbing thing to me was that they never seemed to learn anything from their mistakes. Developers scattered to different companies that had the next great world-changing project starting up and eighteen months later were laid off looking for another gig.

In all honesty, as a developer it wasn’t so bad. I learned early on that the trick to being a happy “coding cowboy” was to never get too comfortable in your saddle. As long as you went into a project knowing that, chances are, this thing will explode on me before it ever sees the light of day, then everything else was pretty good. The pay was great, we were usually using extremely new and exciting technologies, and the atmosphere was almost always electric and exciting. While it lasted.

As I grew older and tired of moving to a new project every year or two, I began to question why projects never seemed to get it right. No matter how large or small, no matter what ex-football player or admiral was on the board of directors, no matter how much venture capital (or, as we called it in the development world, “vulture capital”), eventually the walls started crumbling and we shifted into “death march” mode where we were putting in eighteen hour days and consuming truckloads of Mountain Dew and pizza in a doomed effort to fix the mushrooming bugs and new requirements on an ever-dwindling budget.

These types of projects have been documented before. A book was written about one called “Dreaming in Code” which chronicled the rise and subsequent crash and burn of an open source project called “Chandler“.

The thing that I kept coming back to when considering all of these different project failures (other than “I must be an albatross!”) was that there was some hidden, ticking time bomb in each of them that at some certain point in the project exploded and blanketed everything with chaos. Suddenly, just as budgets were getting strained, deadlines had been missed, and people started getting burned out, everything seemed to just fall apart. Bugs appeared by the dozens, then hundreds, and then thousands, as if they were breeding. Usually about this time (when the first development releases were being tested and put in front of the users), missed requirements were discovered and had to be “grafted” back into the application. This, of course, resulted in more bugs that had to be fixed, and fixing bugs often led to new bugs, and on and on it went until the company ran out of money and dealt with it by laying off their most expensive people first, which also deprived them of their most experienced developers. Soon thereafter, the company would declare the project officially dead and move on to their next world-changing disaster.

What was this mysterious force that seemed to attack a project like a virus and bring it to its knees? How could so many smart people screw up so badly so many times? This problem really bugged me for several years.

In 2008, I was once again looking for work after the last company I worked for, a huge consulting firm that would have been kicked off the New York Stock Exchange for accounting irregularities had it not been kicked off sooner for trading below $1 per share for a solid month, went belly-up. The project that crashed and burned on me in that instance was an internal accounting system that was supposed to correct the accounting irregularities, but it failed to make it out of the gate due to the same old bugs and delays. In this case, a failed project actually did bring down a company of over 15,000 employees…and I was lead developer on the smoking project! This really brought the problem home to me. I had done everything I could conceivably do to make the project a success and thereby prevent the collapse of a huge company and had utterly failed. It weighed very heavily on my mind. I took some comfort in the fact that I had been taken off the project after insisting that we not completely re-engineer it from the ground up a fourth time, but at that point it was pretty obvious that it was doomed anyway. It was technically still on my watch.

While looking for another development job, and questioning my qualifications to be lead anything after the spectacular failure that got my development team and thousands of other people laid off, I was offered a temporary contract job at a large shipyard to do some “systems engineering” work. I had no idea what they wanted me to do, but I aced the interviews and they felt confident I was qualified. At the same time, I was wondering if maybe I should get out of the software development business anyway, given my track record of killing off otherwise viable companies. Maybe some “systems engineering” or hamburger flipping was just what I needed. So I took the job.

As it turned out, I went from the frying pan and into the fire.

Part 2: CYA Agent

In our example project, the very important ID property was omitted from the Project Integrity Items (PIIs) that were created. I will now demonstrate how to add those IDs in as well as other supporting information that will make the project integrity stronger.

As we left it, the project in TreeLine looked like this:

Several PIIs with different Properties

Our first order of business is to add unique identifiers to each PII so that they can be tracked across versions without the risk of misidentifying them. To add a new property to the first PII, simply right-click on the Language property and select Insert Sibling Before. This will create a new property immediately above Language.

Inserting the new ID property

To emphasize the point that a sequential number is not a very good way to handle unique identifiers, this example will use a made-up format of three letters plus three numbers. In a real project it would be advisable to use a standardized unique identifier such as the GUID or UUID format.

Once this property is created, it can be copied and pasted for each PII and a unique value substituted for each one:

ID Property added to each PII

Another piece of information that would probably be useful to track is an effective data for each version. This represents that date that the version was created or put into use. Adding an Effective Date property to versions is just as easy as adding the ID field was…simply right-click the version, then select Add Child:

Adding a property to a Version

Just like last time, a new child item was added of the type “PII”. However, in this case, we don’t want to add a PII;we want to add a Property to the Version. This is no problem. To change the new PII to a property, simply right-click and select Set Item Type , then select “Property” from the list that is displayed. This will change the new item from a PII to a Property:

Change Item Type

This will instantly convert the PII into a Property, which we can then name “Effective Date” and add a date value for:

Effective Date added for all Versions

As the above example shows, items can be nested down more than one level. In fact, with TreeLine, they can be nested as many levels as desired. This is one of the reasons that TreeLine makes such a great tool for illustrating PIM methodologies since PIM projects can possibly have many levels of nesting.

Note in the screenshot above that some nodes have been collapsed by clicking on the “-” beside the PII name. Any node in TreeLine can be collapsed in a similar fashion, allowing easy viewing of items that are not necessarily side by side in the tree view control.

I would now like to show you how easy it is to create a new version of a project. In this scenario, a version has been developer tested and has been sent to testers for regression testing. At this point, any new changes must be made to a new version of the application code. This can easily be reflected in the TreeLine project with simple copy and paste procedures identical to the ones used previously to copy and paste properties and PIIs.

To create the new project version, first collapse the initial version all the way down by clicking on the “-” beside the version name (0.0.0), then right-click on the version and select Copy.

Copying Version Structure

Then right-click on the Widget Manager project and select Paste:

Pasting the copied Version

Another version 0.0.0 will be added. Clicking on the “+” beside it will reveal that it did a complete copy of every item in the original version (i.e. a “deep copy” for you programmers)!  This copy is completely independent of the original and can be edited to reflect the changes made between versions. For the items that did not change, no editing is needed:

New Version Created and Edited

This is a very powerful tool to manage multiple versions of a project. With a little thought, one can imagine many ways that this deep copy and paste technique can be employed to reduce redundant data entry throughout a project under PIM control.

Set Me Free!

One problem with using any tool is that, while the tool may be useful, it is often difficult or impossible to use the data across different tools. For example, a software development shop may be using automated build tools to create builds and deploy them to test environments, and developers may rely on a reporting tool that notifies them when any source code versions change.  Without some sort of interface or export features, the data becomes locked in and inaccessible to other applications that may need them.

While I am not advocating TreeView as a large scale PIM solution, I do see many, many elements in it that lend themselves to this type of methodology. Among those is its ability to export all data in standard data formats. Using this feature, the PIM administrator could easily produce XML data files that other tools could then read and manipulate for related uses such as automation and reporting.

Exporting data is easy with TreeView. Data can be selectively exported by node (selected node plus all children) or as a complete file.

In this example, we will export the data in XML format, but exporting to CSV, HTML single file, HTML Directory Tree, XSLT, and other formats is possible.

First, click on the top node that should be exported. In this example, I will export the entire project (i.e. both versions). To do this, I click on the top node (Widget Manager), then click the File menu, then Export :

Exporting Data

Then select the format that the data should be exported in:

Export Format Selection

Clicking OK will start the export. In this case, it creates a standard XML file of all of the data, shown here in an XML text editor:

XML Output

This file can now easily be imported by any other application that understands standard XML formatting.

Conclusion

I will inevitably get questions on whether or not I would use a tool like TreeLine to do actual Project Integrity Management. I have to say that, as much as I like TreeLine, it is not specialized enough for true PIM. Some examples of where it falls short for this use are:

• It lacks element level validation. This is critical for PIM because data cross-checks must be done to prevent inconsistent versioning, duplicate IDs, and many other potential data entry problems.
• It does not allow “pointers” to items. Instead, each version of a project must have its own copy of each item. This means that if a property is added to one version of a PII, it must be manually added to all other instances of it as well.
• It lacks a structured workflow. Although it makes it very easy to build related groups of items, it makes it just as easy to create unintended duplicates, incomplete entries, and other hard to manage issues.
• Managing data by Copy & Paste is asking for trouble. Most developers have learned to despise “copy and paste” coding because it is difficult to maintain and propagates errors each time it is done. While copying and pasting trees in TreeView is a great timesaver compared to manually entering each item, it is not a realistic approach to managing data reliably.

What TreeLine does though is make it easy to illustrate the concepts that go into Project Integrity Management and perhaps create a basic framework for any future tools that are created to implement this process in real projects.

It is said that a picture is worth a thousand words, so if that is the case, this article is about 12,000 words too long. I hope you learned something and will come back again as we explore ways to manage project complexity.

In an earlier post, I promised to show how basic Project Integrity Management could be done with very basic tools. Although we could go all the way down to using just the filesystem (nested folders, etc.), that is a little too raw for any practical usage. With tools like Excel, you lose the ability to maintain relationships between the items, and with straight word processors such as Microsoft Word you have very limited support for nested items and data field management.

I finally settled on TreeLine for this demonstration because it is capable enough to show how the concept works, it is open source (translation: free) and works on nearly any operating system including Windows, Mac, Linux, and many versions of Unix. Since I use Linux as my primary operating system on all of my computers, this was a real advantage for me. Additionally, the TreeLine files that are created for data storage are in pure XML, a standardized, self-describing data language. What this means is that a TreeLine file I created on my Linux machine will be usable by someone on a Mac or Windows machine without any conversion needed.

For this article, I have made the custom template file that I used in creating the screen shots available for download at the following link: http://www.dbxsoftware.com/cms/wp-content/uploads/treelinefile.zip

Once you download and install TreeLine , simply download the template file zip file, extract the file treelinefile.trl and double-click it to open it with TreeLine. This file will provide you with all of the data elements that I created for this demo without you having to create them yourself, assuming you are interested enough in following along to actually do all of that. Having TreeLine installed and the template file open will make following along much easier.

NOTE: This article does not explain the finer points of configuring and using TreeLine. For more information on that topic, click here.

Using TreeLine for Project Integrity Management

When you open the template file, your screen will appear similar to this:

TreeLine with template file open.

This shows an initial project name Widget Manager and an initial version for the project of 0.0.0.

Before adding elements to the project, consideration should be given to what the various parts of the project are. Typically there will be a User Interface portion (UI), some business logic/server source code, possibly database access source code, and various supporting files such as configuration files and images.

For this simple example, we will use:

• A User Interface Source File
• A General Configuration File
• Two Image Files

Any project in the real world will certainly have more content than this, but these three will serve to illustrate how this tool can be used to manage the pieces and to illustrate how a project under Project Integrity Management is structured.

The template file contains four data “types” or structures:

• Project
• Version
• Project Integrity Item (PII)
• Property

These four elements are all that are needed for any project when PIM is properly implemented. There is some implied structure in these elements that the template file is pre-configured to take care of. They can be summarized as:

Project is a top level (“root”) element. Although project can be nested, i.e. have “child” projects, it is usually easiest to manage each project as its own root level element.

Versions apply to Projects and PIIs All Projects and all PIIs will have a version. Projects will usually eventually have multiple versions as changes are made to the original codebase.

Projects, Versions, and PIIs can all have no properties, or can have as many properties as needed. Properties can differ between PIIs and between versions. In other words, just because version 2.0.1 has a property called “Status”, version 2.0.0 is not required to have that same property, although it may.

Additionally, the following general guidelines are useful in understanding the structure:

• Projects can Contain Projects, Versions and Properties.
• Versions can Contain Properties and PIIs
• PIIs can contain versions, properties and other PIIs.

This, no doubt, seems to be rather circular, but as the following example will show, it makes sense.

To start off a new project, it should be named and a initial version should be created. In the template, those steps are already done. When you open the template, you should see the following screen:

Basic PIM Template in TreeLine

Clicking on the “+” beside the project name (Widget Manager) will expand the view to show a version 0.0.0 below the project name.

Start out by adding a new child node to the version by right-clicking on the version 0.0.0 and selecting Add Child

Adding a child element

Because the template is already configured to default to the most likely child node type, a new child node with the type of “PII” is added when the parent is node is a “Version” node:

Adding a new PII

As soon as the new PII node is added, you can edit the name and description fields on the right side of the screen (shown already edited).

Now that we have a project, a version, and a PII to manage, we need to know more about what it is. Although the description gives a general idea, it is very likely that additional information will be needed to track this particular item. That additional information is added in the form of Properties.

To add a property, right-click on the newly created PII and select (you guessed it) Add Child.

Notice that this time, instead of the icon for a PII (the small notepad), the new item was created with the icon for a property, a small blue sphere. Also notice that because this is a “Property” type item, instead of “Description”, it has a field titled “Value” on the right side of the screen:

Adding a Property

The important thing to remember about a property is that it can represent nearly anything that is needed for tracking purposes. For a source code file, a good thing to know is the development language it is written. Therefore, this property can be named “Language” and the value can be set to “C#”.

Filling in Property Value

This process can be repeated for as many properties as needed. Once the new PII is complete with all of the needed properties, a new item can be easily added by copying the original PII and pasting it as a new PII in the 0.0.0 Version folder:

Copying an existing PII

Pasting a copied PII

This creates a new PII exactly like the copied one.

Duplicated PII

It is then easy to edit the existing properties to match the second source file. This process can then be repeated as many times as needed to get the initial set of PIIs into the project version. In the following screenshot, all four of our example PIIs have been created and edited.

Several PIIs with different Properties

Some interesting points to make about the above screen shot:

All of the PIIs are child nodes of the version 0.0.0, but each item has its own version as well. This follows the concept of Project Integrity Management with states that all Project Integrity Items must be versioned. This is important, because most source files and documents are versioned by the software development environments that are used to develop them. Including the version with the PII guarantees that any changes in the software development environment versioning is accounted for in the project.

One thing that may have gone unnoticed in this example is that, according to PIM rules, each PII must have a unique identifier. These PIIs do NOT have a unique identifier associated with them. In my next article, I will show you how to easily go back and add information that may have been forgotten or omitted in the original setup of the project, as well as some pretty amazing tricks for versioning entire projects and exporting them in standard data formats.

“You can’t solve a problem with software that you can’t solve on paper.”

In other words, software can only solve a problem if that solution has already been figured out logically and codified in the logic that the software operates on. For this reason, my software requirements gathering sessions are always done with nothing more than paper, a whiteboard, and the people who know the processes in a room together. Crazy, huh? A software developer who will only use paper and whiteboards. This approach does two things. It forces the people who already have the knowledge of how to solve the problems that your software is intended to solve to break the processes in their brains down into individual steps, and more importantly, it exposes the gaps that exist in their own manual processes. Until those gaps are filled and they can explain how they do the task manually, the software solution cannot exist.

The reason I bring this up is because the software that I have seen for “Configuration Management” is woefully inadequate for the task. People who create this software clearly do not understand the complexity of the problem they are approaching. How can I make such a bold statement without actually talking with them? I look at the tools they use. As I talked about in the article titled “When all you have is a hammer…”, they are obviously attempting to solve complex problems by using whatever tools they have lying around. A prime example of this is the ubiquitous Treeview control.

If you have ever used a modern computer, you are familiar with the TreeView control. It is everywhere. Most operating systems make wide use of it, particularly for navigating their filesystem. In Windows Explorer (or “My Computer” to many), the TreeView allows users to organize and find files that are represented as icons in folders. When a folder is clicked, the treeview expands to show folders and files within the selected one.

It is a cozy, well-understood metaphor.

Like most metaphors though, it is limiting. I will explain. Have you ever really thought about the whole “folders and files” metaphor? How well does it map to something real? That is, after all, the whole purpose of metaphors…to access knowledge you already have and apply it to a new concept. Ask yourself the following questions about the Windows “filing cabinet” metaphor of folders and files and consider whether or not the metaphor is doing more harm than good:

• Do you really keep your printers in a file folder named “Printers and Faxes” in your filing cabinet at home? Do they fit?
• Do you also keep your expensive camera in your file drawer with your scanner?
• When you want to schedule a task, do you have a folder for that in your file cabinet?
• Do you have a very large file in your file cabinet called “Administrative Tools” with various tools in it?
• Do you keep folders inside folders..inside folders…inside folders…inside folders…with one document in the innermost one? Why would you do that?

Of course, the answer to these questions is almost certainly “no”, but what does that prove? Everybody understands what it means, so why bring it up? Because the only reason “everybody” knows what it means is because the same “everybody” had to unlearn something for it to make sense. Consider this example.

If you were to sit your grandmother who had never used a computer down in front of one and start explaining to her that everything on the computer was stored in files, just like her filing cabinet, she would intuitively understand it at some level because at some point in her life, she learned how to use a standard paper filing system. As you progressed through the various folders (“My Documents”, “My Pictures”, etc. she would probably have little trouble understanding the metaphor, although she may think it odd that you keep your photos and audio “records” in a file cabinet. But when you clicked on a folder called “Printers and Faxes”, what would she expect to see there? If she is like my grandmother, she would expect to see owners manuals for her printer and her fax machine, and maybe the receipts from when she purchased them. She would certainly NOT expect to see an actual laser printer there, any more than she would expect to open a paper file and find one crammed into it.

Here is the problem with metaphors. While they save a lot of learning up front by allowing someone to transfer knowledge that they already have to a new concept, they soon start taking those gains away in the form of learning the ways that the metaphor is bent to apply to the new concept. Files on a computer are LIKE files in a filing cabinet, but they are not REALLY files in a filing cabinet. Since they are not identical, there must be differences, and those differences are where we get into trouble. The more the target of the metaphor (in this case, the computer) strays from the actual item (a filing cabinet or physical desktop), the less useful the metaphor becomes, sometimes with disastrous unintended consequences.

When I was a teenager, I had thousands of hours logged riding my bicycle. I would literally ride miles per day on asphalt, dirt roads, gravel, and even through the woods. I could ride my bike across a large log crossing a stream without falling off and could jump over progressively higher ramps until I eventually crashed and bled (that is how we knew we had reached our limit). In short, I was a good bike rider.

A friend of mine had a cousin who got a motorcycle that he was very proud of. It was a small street bike in pretty good shape. He brought it over to my friend’s house to show us, then insisted on taking my friend for a ride. I followed along behind them on my bicycle to a new highway that was being built. There were no cars on it. Heck, there wasn’t even any asphalt yet…just miles and miles of red dirt. He asked me if I wanted to “take ‘er for a spin”, which I took to mean riding on back of it like my friend had. To my shock, he jumped off and said “go ahead”.

“I’ve never driven a motorcycle” I said. “I’ll probably wreck it.” I was telling the truth.

“Oh, there’s nothing to it!” he insisted, “Its just like riding a bicycle, only you don’t have to peddle!”

I got on and he showed me the throttle (I didn’t have one of those on my bike). Just keep it in first gear and ride a little ways up”, he said, grinning, “it’s simple!”.

Against my better judgement, but at his insistence, I started out by giving it a little gas and suddenly I was putting along at a few miles per hour. Cool. I did a wide turn around and came back to him. “Go again!” he shouted, “and give it some gas this time!”. I did. It felt great. It was much better than being on a bicycle! I cruised about two hundred yards and was really getting the feel for it when I hit a patch of sand.

This happened to me all the time on my bike, and I knew just how to handle it. I applied the brakes a little to slow down and regain control. Surprisingly, this made things worse and I started to seriously swerve towards the embankment. Although I was only travelling at fifteen miles per hour or so, it suddenly seemed to be way too fast. I decided to just stop and start over, so I applied the back brake and front brake heavily at the same time and coasted to a nice smooth…arc…over the handlebars and onto the dirt road. The motorcycle flipped over beside me and sputtered out. I lay there wondering if I had broken my neck on my first motorcycle ride.

By the time my friend and his cousin came running up, I was on my feet surveying the damage. I was scraped up, but not badly. His motorcycle on the other hand was pretty messed up. The turn signals on one side were hanging by wires and the front wheel was having serious disagreements with the handlebars on which way was forward.

“You crashed my bike, you idiot!” he yelled.

Motorcycle controls

“I apologized profusely as he put the front wheel against a nearby tree and twisted the handlebars back straight, then sadly examined his broken turn signals. “My brand new bike! What happened?” he asked.

I pointed out the swerve marks in the sand leading up to the tragic spot. “I hit some sand and lost control. I don’t understand what happened. It tried to stop and it flipped. It’s like I hit a hole in the road, but there isn’t one there.” I blabbered, completely puzzled at what had happened.

“Why didn’t you just hit the brakes?” he asked.

“I did…that is the weird part. As soon as I hit the brakes, it flipped.” I explained.

“Show Me”, he said.

So I got on the bike and applied the brakes again, just as I had. The brake felt spongy. “I think your back brake is messed up”, I said.

“Brake?”, he yelled, “you really ARE an idiot! That isn’t the back brake…it’s the CLUTCH!”. He pointed to the lever on the left handlebar that I was squeezing. “You put on the front brake and the clutch! No wonder it flipped!”

“That’s the clutch?” I said meekly, “On my bike it is the back brake. You said that it was just like riding a bike, only without having to pedal. So I thought that was the back brake.”

“Oh. Yea”, he said, much more calmly. “I forgot to tell you. The back brake is down by your foot”.

Because he offered up a metaphor that I so completely understood, I was overly confident in my ability to control the motorcycle. When someone says that something is “just like” something else, it is always wise to press them further, because if a motorcycle was “just like” a bicycle, it would be a bicycle, not a motorcycle. Instead, it was “very similar with some extremely important differences”.

As this story illustrates, metaphors give, and they also take away. At some point, all metaphors break down and differences appear.

The Tired Desktop Metaphor

DOS Command Line

Early in the days of computers, the concept of a “desktop” was popular. This did not mean “a computer that could sit on your desk”, but rather it was a metaphor for using an otherwise extremely intimidating device. Anyone who has used Unix, Linux, or DOS can attest to the fact that, although the command line is powerful, it is extremely hard to grasp in any meaningful way. The desktop concept provided an abstraction from the true nature of the beast and mapped existing knowledge to operations in the computer. For example, instead of “text files”, you could work with “documents”, Within those documents, you could cut, copy, and paste; these were concepts familiar with nearly all office workers of the time since publishing was a manual process. If you wanted to put a picture in a paper document, you literally cut it out with scissors and pasted onto the paper.

Initially the desktop metaphor was a real hit. It let people jump right in and start using a computer without any real computer knowledge. It was such a hit, in fact, that Bill Gates created a fortune on the concept with the introduction of the “Windows” operating system, which leaned heavily on the desktop metaphor.

Windows 3.11 Desktop

With the Windows desktop, the user was presented a “desktop” with different tools on it, much as a real desk of the 1980s may have on it. It did not have a stapler (although I always thought it should…to stick files together as a group, but that came later as Windows Briefcase”), but it did have a clipboard, a clock, a calendar, and various other tools that made sense in the context of a desktop. And most importantly, it had a file cabinet called “File Manager” whose icon was a yellow file cabinet.

Over the years, as the capabilities of the computers and operating systems evolved, Microsoft struggled to keep the desktop metaphor. Odd items started showing up that most people wouldn’t really have on their desk such as a mailbox, other people’s computers (Network) and a strange piece of machinery that was apparently attached to the front left edge of the desk called a “Start Button”, which had absolutely no discernible function in the real word.

Windows 3.1 File Manager

This is not to say that these changes were not improvements, as they most certainly were. They made using a computer easier, more efficient, and in many cases, all around better. But at the same time, they stretched the desktop metaphor to the point that it is no longer recognizable a metaphor. Looking at the Windows 7 desktop, there is obviously complete abandonment of the original idea of mapping tasks on the computer to tasks in the physical world. Yet the metaphor lives on. Why? Because its what people know. But not the original metaphor. Someone who cut their virtual teeth on Windows XP would have no better idea of how to operate Windows 3.1 than someone who had never seen a computer. In fact, they would be more frustrated because they have learned the extensive metaphor “breakers” and accepted them as common knowledge.

Windows 7 Explorer

Meanwhile, we are still trying to organize our filesystems into file cabinets and folders. Even the name filesystem is left over from the metaphor. Now instead of providing us free knowledge, the metaphor is holding developers back. In conforming to the age-old desktop (and more importantly, filing cabinet) metaphor, they are actually limited in what they can do with software. The metaphor has come back to collect its dues.

This doesn’t just happen in the computer world. I was unfortunate enough to be hired by a huge consulting firm as a senior developer in their brand spanking new software development center (as in “at the center of nowhere) . The place was designed to provide work areas for 200+ developers, but at the time I was hired, there were fewer than fifty, so there was a large area of the operations floor we referred to as “the back forty” that was not being used. We would sometimes walk through it to get to a server room on the other side of it, and one day one of the young developers in my charge stopped and surveyed the huge area filled with cubicles, shelves, whiteboards, etc. but no people.

“Man, this is nice!” he said, looking around, “This place is really big! What are these big metal things?” He was referring to a row of no less than fifty large vertical filing cabinets that lined the central walkway.

“Those are filing cabinets,” I said, opening a drawer to show him.

“What are they for?” he asked.

“Filing papers”, I replied, amazed at his ignorance.

“What papers?”

“Hmm…you’re right. I don’t know.”

It dawned on me that whoever had designed the development center had somehow calculated how many filing cabinets would be needed for 200 developers. Developers who were not familiar with filing cabinets and would certainly never use them.

The Metaphor Lives On

This trend has continued for a very long time as people continue to discover layers of complexity under seemingly simple concepts. Something else is taking place as well, something that is the exact opposite of finding complexity in simple concepts: the discovery that complex concepts have simple underlying processes. For example, although there are many, many types of materials in existence, they can all be broken down to molecules, and then to a finite set of elements. These elements are made up of a few basic parts called atoms which obey a fairly simple set of physical rules. These can be broken down in

Caffeine Molecule Model

to smaller components that are even simpler in structure. What this means is that even the most complex life

forms on this planet can be understood by looking at them as a system of components, not as a whole.

The human body is extremely complex, but can be better understood when divided up by system. The digestive system, although interconnected with the rest of the body, can be understood according to its specific functions, as can the circulatory and nervous system. Each of these systems can be broken down into smaller components (i.e. heart, arteries, veins, capillaries, blood, etc.) and each of those components can be further understood by breaking them into smaller components (i.e. red blood cells, white blood cells, haemoglobin, etc.).

From this example, it is easy to see how the process of simplifying the incomprehensibly complex human body into smaller and smaller parts helps us understand not only what those parts do, but how they contribute to the entire biological processes that result in a living, breathing human. This is not to say that we understand everything about the problems that science examines. We don’t know exactly what causes most cancers, we don’t know exactly how gravity works, and there are many, many concepts yet to be understood, but there is no denying that we have learned much more about complex systems by breaking them down into simpler components.

This concept is largely what Project Integrity Management is all about. Projects are extremely complex things. When a software product is completed, it typically consists of thousands of individual files, hundreds of thousands of lines of code, and millions of individual processes that must happen in exactly the right sequence. Managing something this complex is impossible to do effectively without simplifying it in some way.

Most projects already do this to some degree. Database code is typically developed separate from application code, and application code is typically developed in “layers” such as in the ubiquitous “Model, View, Controller” method of

Typical MVC Layout

organizing code. This makes the jobs of developers, testers, and everyone else involved in the project easier by providing a logical separation of components, each having a specific function in the application. Even application code is further organized by intended function with the use of libraries and namespaces.

This organization only goes so far though. With a typical software project, there is nothing controlling how changes in one part of the application code will affect another part. There are no mechanisms to insure that a particular piece of application code is always run with other compatible pieces. Because of this, what starts out as a fairly organized project soon breaks down into a disorganized, bug-ridden nightmare. Something more is needed to insure that all of the different components are working together as intended. Something must insure that changes to the database code are accounted for in all of pieces of the application that rely on it. Something must insure that testing is done against the proper version of each and every component to insure that the testing is valid. This is where further simplification is needed.

The concept of the Component Integrity Item is similar to the concept of the atom. It is a fairly simple structure that can be uniquely identified, carries with it certain properties, but can be treated generically. This means that each component in a project will have a Component Integrity Item associated with it, and this CII will have a unique identifier that is used only for that particular component in that particular project. Additionally, it has a version associated with it to account for any changes that will occur to the component over time. These are the minimum requirements for a CII. Additionally, they can carry with them information about specific components such as the file name, the format, and other relevant data. These data items are referred to as “properties”, and each CII can have as many properties as needed.

In practical use, this allows us to manage complex arrangements of components in a very simple way. The following simplified example traces a CII from its most basic form up to the full application level. Note that the CII that started out as a single item at the lowest level typically becomes part of a larger CII as it moves up towards the full application. Those aggregate CIIs also become part of larger components, until they reach the very top…the project…and ALL CIIs are part of the project CII.

Project (ID: 1)
---------------
             Application Code (ID 214)
             -------------------------
                                    ...
                                    Database Library (ID 784)
                                    -------------------------
                                                          ...
                                                          Data Retrieval (ID: 1245)
                                                          --------------------
                                                                  MySQLData.cs (ID: 2451)
                                                                  MySQL.ini (ID: 2452)
                                                                  MySQL.dll (ID: 2453)

Note that in this hierarchical view, the continuation mark (…) is included at each level. This is to indicate that there are many more items that fall in that level that are not listed in this example. If you were to include them, and trace them down to their individual components as was done with the Data Retrieval library, you would find similar structures under each of them.

Also note that the ID numbers do not follow any particular sequence. This is intentional to show that the structure does not depend on the actual unique identifier’s format or sequence, only the fact that it is indeed unique within the project.

A typical Mandelbrot Fractal Plot

In many ways, this concept mirrors the concept of Fractals, a mathematical construct that mathematician Benoit Mandelbrot proposed as an example of how simple structures can result in complex structures when combined in a specific way.

Simpler examples of this same concept can be found throughout your everyday world. Electronics are made up of thousands of microscopic transistors integrated onto microchips. Extremely complex and realistic animations are made by combining millions of tiny polygons, electrical systems are made up of simple wires, switches, and sockets, and the list goes on and on.

By looking at a project as a collection of simpler items, we will be able to move away from managing complexity and towards managing simplicity, which is much easier to do. By managing the simpler aspects of the project, we create the end result of managing the complexity of it since that complexity arises from the interrelationships of the simple components that make it up.

In my next article I will show how general software tools such as Excel, Access, or even text files can be used to manage project complexity.

When that was met with an offer to transfer some of the list items to him, he bowed out by saying "It won’t do any good…you have lists of your lists."

The more I thought about that statement, the more I realized that, boiled down to its absolute simplest form, that is what the concept of Project Integrity Management really is. Lists of lists. Instead of lists of tasks to get done though, they are lists of components of various environments.

A list by itself is not extremely useful. It can help us make sure we remember the eggs at the grocery store or to pick up the jacket from the dry cleaner, but if more than a dozen or so items are added to it, it becomes unwieldy and, in my wife’s case, somewhat depressing.

Now that I think back, perhaps is isn’t a weird coincidence that she gravitated towards a similar concept when dealing with her lists. I recall several years ago when we were building our house she had a list that would make even the most hard-core procrastinator cringe. Looking at this endless list of items like "Install Windows", "Wire House", "Install Bathroom", etc. was enough to make me want to just sit down in the middle of the slab (note: add "install flooring" to list) and cry. As a method for dealing with this eternal list, I suggested breaking it into smaller lists and make them line up with the project dependencies. For example, I could not "Install the Bathroom" until "Wire the House" was done. Of course, Wire the House had many smaller tasks such as "Install Load Center", "Install Grounding Rod", etc.

These items caused another dependent list to be created for the purchase of the needed supplies such as Load Center, Breakers, 2.5 Miles of ROMEX wire, etc. etc.

After going through this process, we ended up with a large pile of shorter lists, but to her organized, list driven mind this was a step back, not a step forward. Now things were not all neat and tidy in one big list, but instead scattered throughout dozens of lists. I was on the brink of losing the battle and falling back to the original list if I didn’t do something quickly. Out of desperation, I suggested a list whose only purpose was to organize the other lists.

After much coaxing, explaining, demonstrating and a dose of Xanax, she agreed to give it a try. I then compiled a list of lists in the order that the big things needed to be done. So first on the list would be "Wire House", followed by "Install Plumbing", followed by "Install Bathroom Fixtures". Each of these items had a corresponding list that had items such as "Purchase Wiring Supplies" (which in turn pointed to the shopping list for these items), "Purchase Plumbing Supplies’, etc.

Looking back, I now see that this system was probably a first attempt at the ideas that eventually became Project Integrity Management. By creating a hierarchical "List of Lists", I could concentrate on the portion of the overall picture that needed to be worked on next without worrying about the rest. If I ran into a delay on one list, I could leave it in a known condition and move to another list until the obstacle was cleared, then come back to it. As I completed the tasks on the smaller lists, they were marked of on their list and once the full list was complete, the item that referred to it was marked of the main list. For example, in installing the plumbing, I would mark off "Install cold water pipes", "Install hot water pipes", "Install water heater", "install drain line", etc. etc. until the list was complete. At that point, I could go to the main list and mark off "Install Plumbing".

Believe it or not, we did actually finish the house and as far as I know, everything got done. It didn’t always go according to plan, but with the flexible "List of List" (LOL!) system, we were able to manage the complexity of the project in small chunks. At a basic level, this is what Project Integrity Management does. It organizes and manages the extremely complex task of building a software product, passing it through multiple environments, and shipping a clean, stable product in the end. Isn’t that all we really want from a project?

M@

This is just plain silly!

The term environment is used in a variety of ways. Recruiters speak of a company’s “work environment”. Psychologists talk about a healthy home environment. Environmentalists  talk about the natural environment. What all of these concepts have in common (besides the word “Environment”) is the concept of the shared space and components that surround us.

In software development, when we speak of “environments”, we are referring to the complex relationships between a set of components and software. A typical project will have one or more environments each for development, testing, and production. This includes the things normally thought of by development teams such as the operating system, the database management system, the database itself, and the supporting components such as shared libraries and various supporting and configuration files. Some other things that may not normally be considered as part of a development environment are the hardware components, peripherals such as input devices, scanners, printers, and external storage.

Most projects spend a lot of effort maintaining their source code using version control software, but how many projects put that level of control into place for their environments? Why should they?

Today I am discussing why it is critical for any project that intends to succeed (which I hope covers pretty much all of them) to actively control and maintain their various environments. Before making this case though, it is important to define what is meant by the term “environment”.

As stated in an earlier article, the purpose of Project Integrity Management is to maintain a state of “known goodness” throughout the development process. In terms of environments, this means always knowing positively that the environment that software is being tested in is the right environment…that all software  versions are right, that all configuration settings are known, and that any device or component that the software touches is as it should be.

On the surface, environment management seems fairly simple. By setting up three application servers and three databases, three different environments can easily be created. We will call them Dev (Development), Test (Testing) and Prod (Production).  Ideally, software will be developed in DEV, testing in TEST, and put into production in PROD. Since these environments were identical when they were created, they should remain identical throughout the lifecycle of the software product…right? You wish.

In reality, things are not so clean. For example, timing is an issue. The Widget is developed in the DEV environment, then handed off to testers to validate it in the TEST environment. It will take several days to a week to completely test it, and in the meantime, development continues on the next set of development tasks. During this time, developers realize that the workflow library that they originally used has a bug that they only discovered after the initial effort was turned over to test. To correct this bug, they must upgrade the workflow library from Version 2 to Version 3. However, in order to do this, they must make a slight change to the way the library is used, so they cannot upgrade TEST while the testers are still validating the last build. Now the environments don’t match.

In a more common scenario, the DEV database is constantly changing as new features are developed and the database support for them is added. By the time a new release of the software is ready to go to TEST, the TEST database is obsolete.

There is a myriad of other ways that environments can get out of sync, and these subtle differences that are very hard to detect can cause days and even weeks of project time to be lost. I have seen projects with poor environmental control where testers would find bugs, development would fix them, testers would validate that fix, only to have the exact same bug show up again two or three releases later.

These examples are very basic and don’t even consider the complexities that come into play with larger development projects that can have many DEV, TEST, and PROD environments which must be kept in sync at all times. It also doesn’t really address the problems that can occur when there is an intentional difference between versions of the same environment. It can get overwhelming very quickly.

The importance of environmental control cannot be overstated in any project, especially a large project. As we progress, I will provide ideas on how to set up, manage, and ensure the integrity of your various environments. I hope you stick around for it.

M@

NOTE: My apologies to those of you who thought you were going to read an article on how software can save the planet. Maybe I will write on that subject in a later post.