Beware the anecdote

“This one company used our software and they realized a $10 million savings in one year!”

Good for them. Frankly, I don’t (and you shouldn’t) care. The anecdote may be the most disastrous (from the consumer’s perspective, not the sellers) marketing device ever employed – short of outright lying I suppose. The anecdote takes a great story and uses it as the basis for why you should take your hard earned money and buy product X, or Y or Z.

Anecdotes abound. We are a culture of story tellers. The media exploits it all the time. The government exploits it all the time. Just think about the current health care debate. On one side you have the story of the Canadian woman who would have died had she relied on Canada’s healthcare. On the other side, you have a very happy Canadian health care recipient. How can it be that two totally different anecdotes are coming out of the very same system?

It’s a logical fault that we make. We assume that the related experience is the representative experience, which it clearly isn’t. Well, at least not all the time. Consider the “bad” Canadian health care experience. Let’s assume, first of all, that she really would have died if she hadn’t come to the US. We don’t actually know that because we can’t explore alternate realities. She believes she would have died, so let’s take it at face value. Has anyone in the US health care system ever died due to a bad diagnosis? If you answered “no”, I’d like to have some of whatever drug you are on, because that stuff is creating the kind of delusions not seen since the Emperor went out without his clothes on.

Of course there have been bad US experiences! People die here too! But the anecdote is powerful. It’s scary that someone might slip through the cracks. And statistics are oh so dreadfully boring! If Wikipedia is to be believed, the US pays twice as much and yet lags other wealthy nations in life expectancy and infant mortality rates. That doesn’t mean people don’t die in these other countries of horrific, potentially preventable deaths in these other countries – they do. But the odds are better. And even with improved odds of you surviving infancy and living longer, there are scary stories to tell.

People relate an anecdote when it supports their view of the world, whatever that may be. However, they’re doing that because the general patterns of the product, process or service aren’t necessarily an experience which causes you to take action.

I mean, would you be all that excited if I said “if the US adopted a health care system like Canada’s it’d probably be about the same care as you get today, but maybe a little cheaper.” Wow! I mean, wow! If that isn’t a call to action, I don’t know what is!?! People are selling things all the time – whether it is software, a change in the healthcare system, or NOT changing the health care system. If you take that anecdote and relate it to another person, who passes it along to another, and so on then it has served its powerful but potentially destructive purpose. Nobody can remember the statistics, but we all remember a good story.

The next time someone tells you a story about why you should take action, instead of listening intently, ask for the data. Then go verify the data. But most of all, beware the anecdote!

Cyclomatic complexi-what?

It’s been a while since I’ve had the opportunity for a good rant.  Today provided it.

Not that long ago I wrote about the new measurement system QA was proposing - defects per test case – as a proxy for code quality.  Anyway, as part of that effort I had to then go out to all the middle managers and explain our approach and how we arrived at that conclusion.

I’m delighted to be asked to give a talk about statistical analysis whenever I can.  And it’s always a fun challenge when someone else enters the room who believes they’ve got similar or better ability.  I genuinely like having my stats knowledge tested.  But there’s another type who comes to meetings who I don’t like.  People who don’t bother to challenge the analysis but just dismiss the idea as wrong.  Take for example today.

“I have some concerns about using defects to measure quality.”

You what!?!?  That’s like saying, at least in my mind, I have some issues using rulers to measure length.  The best minds in the IT field recognize that defects, typically in the form of defect containment is one measure you cannot live without.  He goes on…

“Quality doesn’t necessarily mean defects.  It could mean cyclomatic complexity or what value the business derives from the functionality.  If we don’t build it well and it is unusable then it isn’t high quality.”

I can feel my face turning red as he pontificates on the subject.  It’s good I maintain my cool.  For one, what the heck is “cyclomatic complexity.”  I can ascertain from the context that it is some measure of the code complexity.  What to even say.  I think two words sum it up.  WHO CARES!  I know, I know, how dare I say that.  Everyone knows that having maintainable code is good for the customer down the road when they request changes.

Let’s be honest.  Option one is the Agile approach where you build only what you need and no more.  If you end up having to add a feature that breaks the design you “refactor.”  If you are doing waterfall, you try and anticipate some future need to reduce the risk of having to redesign, but eventually, your model will be broken and you’ll have to refactor or band-aid it forever.  You’re not going to get away without rewriting parts of the system.  Ever.  The complexity of your system will rise and fall over time.

Anyway, more importantly, and here’s the secret.  In the immediate need, YOUR CUSTOMER DOESN’T CARE ABOUT CYCLOMATIC COMPLEXITY!!!  You can write the crummiest spaghetti code that has ever been authored and if it meets your customer’s need, and they don’t modify it, they’ll never know.  They’ll never care.  Having high cyclomatic complexity might (and I stress MIGHT) be a leading indicator of future cost, but I can’t think of a single customer I ever encountered who said “I won’t be happy unless this product has low cyclomatic complexity.”  Don’t confuse your idea of what quality means with what your customer says.

On the other front, business value is a ridiculous way to look at the quality of what you deliver.  Now don’t get me wrong, you must be unswervingly focused on what your customer wants.  That said, as an organization we have people whose job it is up front to analyze the market need, determine the potential value, gather cost information to assure an adequate CBA and finally specify the product that will solve that need. 

I know you are going to be morally offended, but once it hits systems, you pretty much should be thinking of yourself as an order taker.  Yes, an order taker who can make some suggestions, but essentially an order taker.  You have no control over the value side of the equation.  You didn’t specify the product needed.  The business did.  If the business had some ill-conceived idea of how the solution ought to work that the customers won’t find valuable, it isn’t the responsibility of systems to second guess it.  The entire process of building something isn’t a forum for intellectual discourse.  Eventually someone has to do something and build it!  Creating waiting or rework by debating the request is not lean and if nothing else it’s presumptuous that systems understands the needs of the business better than the business.

Could you imagine on the floor of the Toyota plant some guy down there saying “you know, if we built go-carts instead of cars, I think people would be happier.”  And then proceeded to build a go-cart instead of a Camry?  I sure as heck wouldn’t be excited when they tried to deliver THAT product to me!  There’s always room for innovation on the factory floor, but it’s about ways to better and faster put together the car, not making the car more “usable.”

So sure, ultimately a quality product is something that people want to use.  That’s the “do the right thing” part of the equation.  That’s the strategy of selecting the right product to build.  In terms of the factory floor – analysis, design, code and test – it’s all about tactics “doing the thing right.”

Which brings us back around to our customer, who specifically said (yes, we were smart enough TO GO ASK OUR CUSTOMER!) that defects was their key measurement of the quality of the product.  Funny that cyclomatic complexity didn’t come up…

Today may be just like yesterday should have been

OK, I admit, it’s an odd title for this entry.  Here’s the story.  Some time ago, several years now, a group of people got together and decided what a process organization should look like at our company.  I’m sure it was modelled on other companies ideas.  Then, that same team set about to make it a reality.  They divided the group into three distinct parts – strategic process, tactical process and metrics. 

The tactical process group did targeted improvements.  If someone came to us and needed a process fixed, that was the work of the tactical group.  It was easy to get started on tactical things so this group had an abundance of work.  However, leadership recognized that a purely tactical approach was problematic.  Based on the way the end-to-end process was, the tactical approach was too narrowly focused.  For any given sub-process the assumption was made that the inputs and outputs of that process were basically immutable.  You could only change inside the process so long as you could use the same input and provide the same output.  It’s suboptimizing, but it is better than not optimizing at all.

The strategic group was there as more R&D.  It was supposed to look at the process holistically, take it apart, reassemble it… generally understand it and THEN act upon changing it.  Specifically, the strategic team would determine what the tactical team would focus on. 

Finally, the metrics team was a service group who had the skillset to query various databases and produce charts and graphs and other reports about the process that were needed.

At least, that was the idea of what the group would be.  Then reality hit.  The tactical team had too much work, so they started stealing resources from the strategic team.  It wasn’t long before the strategic team was doing the same thing that the tactical team was doing.  The metrics team had become mired down in manual processes to collect data and produce charts – ironic for a process organization, no?

Anyway, a year or two on, there was only one manager and one team left standing – the tactical team.  The strategic folks had been absorbed, the management moved on and the metrics merged in with some other group who also produced charts.  After all, without the strategic team defining how the end to end process should be measured, there wasn’t much for metrics to do.

Lo and behold, we all get invited to a planning session for the next year.  We’re asked “what are the things that a process organization should do?”  After about an hour or two of people throwing out ideas and whatnot, you could see that the proposed answer looked very familiar.  A process organization, at least so far as we could tell, ought to have a strategic, tactical and support (metrics) arm.  It makes sense.  You need governance.  You need metrics to support that governance.  And finally, you need a handful of tactical people to get their hands dirty, provide facilitation skills for tough business process decisions, etc.

Why does what should be done today look just like yesterday?  And why are we planning for today?  We never did the original plan, so its not like this new plan was informed by our past failures.  Well, other than to say we’re about to repeat it apparently.

On one hand, yesterday’s team structure seemed sensible.  We thought, at the time, that it was the right way to do things.  But it didn’t happen.  The company didn’t support the structure.  So, now we’re at today… do we go back to a failed model and hope it works this time?  Do we just keep doing what we’re doing and leave it at that?

I can’t say that I know the answer, but I will say this: doing the same thing over and over again and expecting a different result is the definition of insanity.  As an organization – your organization, my organization – if you aren’t collecting data and learning from your failures, what exactly are you basing your new team structure/process/whatever on?

The Hawthorne Effect of constant change

I was having a great email chat with my former vice president the other day.  The original conversation was about how a process improvement team should be organized and we got onto the topic of the virtues of a centralized or decentralized model of process ownership.

While centralization seems to make sense, I argued that the change management challenges created by disassociating the improvement work with the teams who had to improve wasn’t worth the added value of defining common processes across silos.  In fact, in our organization, the silos don’t share that many common processes, so even if centrally defined, little would be shared.

From there, as our conversations often do, we wandered over to Agile software development.  My friend is a huge proponent, while I am an outright non-believer (as if you couldn’t tell from my other posts).  And the discussion became about whether one of the main tenets of Agile – that the team decides the process – was a good idea.  In a separate post, I’m going to tell you why no team should ever be allowed to decide the process for themselves, but let’s ignore that for the purposes of this post.

For whatever reasons, valid or invalid, having a team decide their own process makes me uncomfortable.  It may be that I’m a control freak, but something is unsettling about it.  However, over my years I have mellowed from a strict dictator about process compliance to someone who is willing to take some input (the operative word here is some).  So I conceded that I could be supportive of a Toyota-like model, where those who want to improve the process have to prove its value and cause the change to be socialized with the entire organization.  Local sub-optimization is not acceptable.

Well, I don’t know about Toyota and their proof process, but it’s certainly better than the appearance that Agile gives, which is the team collectively decides what the right process is.  No data, just gut decisions.  And then it struck me – the act of allowing change may simply be all you need to create superior performance.  It has nothing to do with the quality of the change!  It’s the Hawthorne Effect at work.  By allowing teams to control their environment, they get the value, the excitement, of trying something new.  And as a result, if they like it, it becomes part of their process.  The value of that particular change may fade.  In fact, it may even be damaging, but offset by the value of enthusiasm.  At least, until the enthusiasm for that change ends.  And then they make another change, which has the same short term benefit, and another, and another and another.  Each change is good or bad, or hopefully evenly distributed, so that once the Hawthorne Effect fades any damage the change might really do is offset by some positive real change that was also made.  Not that you’re really getting anywhere ever, one step forward, one step back.

Of course, I wonder if the constant change eventually loses its magic, the act of improving becoming no longer exciting.  And then of course, all those steps backward will come to haunt you.

Standards for the sake of standards

I’ve been forced to rewrite a large section of this post, since my original optimism about the value of controlling process variation has been dashed to the rocks by a healthy dose of reality.  I have no idea what I was thinking when I first wrote this.

Just the other day I was in a meeting with a bunch of business folks and some people from our team.  Some time back we set out to standardize the requirements gathering process.  Over the course of a few weeks we researched the industry standards and built a model for requirements gathering centered around some very basic ideas like JAR sessions and clear, concise requirements.  Certainly there was nothing that was brain surgery about it.

As we neared the end of our pilots, which people seemed very happy with, we realized we needed to plan for ongoing support for the process.  One of the key features, right or wrong, was that every project would have a trained facilitator for the JAR sessions.  For the sheer volume of projects we do every year, we’re talking about a significant resource expense to have these people on every project.

When we went back to the sponsors and said “hey, we need X millions of dollars to pay for resources over the next 5 years” to support our standard process, it didn’t go over that well.  Their response was, logically, “and what’s the benefit of paying for these people?”

Ah, how much I love it when the rules of the game change.  We’d now gone from standardize to improve.  At least in as much as the improvement had to pay for the resources we were asking for.  What to do, what to do?

I think panic set in at this point.  Standardized was OK with our sponsor as long as it was free.  Since we were asking for resources to support the standard, it suddenly wasn’t appealing to just have a standard.  And now we were being asked to prove that it was worth making the investment.  We’d never planned on it being worth anything, but just that it’d become the basis for future improvements.

The problem is, controlling variation doesn’t add a lot of value when the cost of one project can be offset by the value of another.  In the old process, we used to spend an average of 11.28% of our project money on requirements.  It had a standard deviation of about 7.3%.  In the new process, we’d spend an average of 11.3% of our project money on requirements and it has a 1.7% standard deviation.

Of course, having told you all this, the standard deviation is not meaningful in this case.  Recall from a prior post that the average describes what you’ll encounter over the long run.  When it comes to money, something funny happens.  If you overspend on one project and then underspend on another, it can all come out in the wash.  It’s different from say, building widgets, where if you build it too large or too small that it’s defective.  Money is a different animal and it’s likely that your CTQs will be around average cost, not variation in the process.

Think about it, let’s say you’re offering some service to your customers.  On average it’s costing you $100 to deliver the service.  If it sometimes costs you $10 and other times costs you $190, that’s fine.  As long as it averages out to a profitable number, you’re good.  You can charge every customer something over $100 and even though you’ll lose some money some of the time, you’ll still be turning a profit in the long run.  And if you can’t make money at an average cost of $100, then controlling the variation of the process isn’t going to make the situation better.  You HAVE to move the average.

Anyway, back to the problem.  Since the new process had a slightly (0.02%) higher (read: worse) average cost, in the long run we’d lose money.  Standardizing for the sake of standardizing hurt us.  And even though we settled on what we thought was a good standard, it was not good enough to be worth doing.  We’d actually lose more money by controlling the process than by letting it run wild.

So there you have it, standards for the sake of standards is not a universally good idea.  Sometimes chaos turns out a better result than controlling it.

You are offering me nothing

A turn for the worse when it comes to my post about process or process governance.  I finally heard about the first recommendation for the development process.  It was a recommendation for the system design template.  And the recommendation, never mind the template itself, was garbage!

“We’re going to provide a table of contents,” they said.  A what?!?!  I mean, I know what a table of contents is, but how does someone, much less some group of people, decide that a table of contents (a general outline of the document) would be considered a good new standard?

There is a big difference between “doing things right” and “doing the right things.” The first is tactical thinking, and the second is strategic thinking; the first is management, and the second is leadership

– Terry “Doc” Dockery, PhD

A strategy is a must have.  I’m not arguing against strategy.  A table of contents for a document is a strategy.  It covers the basics of what needs to be done.  It is the “doing the right things” part of the equation.  Alone, that is NOTHING!  It is worthless!  It is an idea that has not be executed upon.  If you are designing a process, you can’t put together a table of contents for a document and tell me to go figure out the rest.  The devil is in the details.  The other half of the equation is “doing things right.”  Where is that in a table of contents???

We already work for an organization that gave me a strategy – a suggestion, at best – of how a system design should be done.  You can tell me (that’s the royal we, referring to all developers in the company) what things I should be doing, but if you give me no direction on how to do it, don’t expect much of the results.

A good process has to have a strategy (a general idea of what the right things are) AND tactics.  You can’t leave out the process flow, the templates and tools, errorproofing, and a means to objectively inspect the work and call it a process.  Otherwise, it’s just an idea.  Not that there’s anything wrong with ideas, but how I choose to execute on an idea is going to be very different than the guy next to me.  It’s probably not what you intended.  If you are going to offer me a process, offer me the whole process.  If not, you are offering me nothing that I can use.

Process or Process Governance First?

This might be an issue of which came first, the chicken or the egg, but I’ve been wondering about it for a couple weeks now.  If you are building up a new process (or set of processes) for an organization, should you focus first on defining the processes or on the governance of a yet to be defined process?

I wonder this because I was sitting in a meeting the other day with a group of IT professionals who were trying to put together a new set of development practices.  On the to-do list were processes for all kinds of things, including a few governance items.  Prior to the meeting, we had been asked to rank all the items in order of preference.

Most people’s lists were similar, except mine.  Almost everyone wanted to put into place a set of practices on how to do the development, while very high on my list were the measurement system and governance components to watch over the process.

Of course, some part of me thought that I might be totally out of touch with the real world.  Of course, it makes sense to define the processes for actually doing the work, I thought.  But then I wondered if this was just a fundamental difference between how I think about things and how most people think about things.  Most people are not, in my opinion, particularly analytical.  Most people like to act, so it makes sense that they’d want to define processes closest to the work getting done.  After all, if you can’t actually take action, at least you could work on something that would enable taking action.  Me, on the other hand, I prefer to ponder things, so setting up the governance and metrics so that I could collect data about whatever we did decide on doing would be important to me.

If you had to choose, and couldn’t do both, which would be better?  Define a process that you lack complete ability to govern and measure or be able to enforce and measure a process or set of processes which is only partially defined?  The easy answer is to do both, but I’ve taken that option off the table.  Clearly there is no value to governance of nothing.  But what about governance over a small something?  If you are deliberative like I am, then wouldn’t you rather be able to manage what you have and then use what you find out to add to the process? 

On the flip side, what’s a defined process with no compliance to it and no ability to figure out how it is working?  If the results aren’t what you like, you can’t do anything about it.  In fact, you can’t even be sure the process is or isn’t at fault since you’d have no clue as to whether someone was using it.

Me, I’d rather be able to correct and improve the situation that I’m in rather than just imagine it is being followed in earnest.  Maybe that’s just me, but I often wonder when I find myself in the vast minority if I might be on to something.

How to gut process improvements

I sat through a very long, very boring meeting the other day on a set of process improvements.  In the room were about a dozen practitioners of software development.  All of them are experienced developers and in that capacity probably capable people.  That does not make them good process people.

For one, there are about a million ways to do software development in a capable manner.  It’s great that there are so many options for people, but it is unreasonable to think that a group of people with such disparate opinions are going to come together and decide on a standard for the entire company.

The problem comes from the fact that there are so many good opportunities to do software development capably. All of the methods are acceptable, and all of them produce about the same results.  If no methodology stands out above the rest, then why change?  It’s a fair question to ask every person in that room.

That wasn’t the point of my post, however.  The point was to show how all of those good methodologies can be combined into a poor compromise.  While each practitioner had assembled a hodge-podge of mechanisms which resulted in software that was on par with his/her peers, they didn’t understand why they got the results they did.  I say hodge-podge because I don’t want to suggest that their decisions were deliberate.  For the most part, whether they followed Agile or strict waterfall or spiral or whatever, they probably learned it from someone who understood far better why they were using that process.  The particular folks in the room were just mimicking the people they learned from.

Oh, how the speculations flew about the room as to why one methodology was working well.  “Let’s take this feature” or “how about that feature” and suddenly we have a proposed list of process features to become a new standard that is probably collectively worse than the individuals.

And what about the rest of the features that each methodology had?  We were making “progress” when the team agreed to defer dozens of aspects of each…

Yes, the team had been tasked with assembling a list of features to become the new standard.  But when did it become acceptable to defer large sections of the process design?  The absence of any robust debate gave away the lack of understanding on how any of the processes truly worked.

Sure, we got through the meeting, but at what cost to the company?  I wouldn’t want to adopt the end result.  As far as I can tell, it is inferior to all the alternatives we started with.

If you want to gut process improvement work, allow a team of well intentioned individuals to assemble the process themselves.  If making processes were easy, companies wouldn’t achieve a competitive advantage from doing things well.  And doing things well is all about the process, isn’t it?

Not invented here, not used there

So here’s an interesting conundrum. Let’s say that I’m tasked with standardizing some processes for my team (never mind my rant about the appropriate scope of process standardization) and let’s say that another team I know already has a fully formed set of processes I could adopt.

It seems to me, assuming that it is otherwise appropriate to do so, that I should take this other team’s processes and simply not reinvent the wheel right? Well what if it turns out that the other team has a completely specified process that they don’t use?

Yeah, ok, so my first question is why did they write down a process they don’t execute? Obviously, this is an adoption issue. Some smart people (and I knew these people and definitely think they’re genuinely smart) decided to set up a standard process. This is a good thing. But, they lacked the change management capabilities (whether it’s a really good sales pitch or adequate command and control) to make it a reality.

On my end of things, how do I make an objective evaluation about this process? The most logical option when deciding whether to adopt a process would be to look at that processes’ capability to deliver results that meet your CTQs. But a process that is not executed has no data to support its goodness.

Unable to put my hands on any data that says the process is good, there’s but one course of action in my mind – dump it into the “not invented here” pile and ignore it.

Wouldn’t it just be great if we could take completed processes, like CMM or Agile or RUP or whatever and just use them with success? Ever notice how all these processes come with a complete lack of supporting data about their capability to deliver results? There’s always the big end result “50% decrease in time to market” or “66% increase in defect containment” or whatever, but they never actually talk about the correlation between the result and the process activities. People wanting you to adopt a process are selling something, so if the story is totally rosy one can bet that some data has been conveniently omitted.

“Not invented here” is not a great policy, but I’m sure that “invented (but unproven) there” does not make a good enough case to start using someone else’s stuff.

By the way, when I set out to start writing this blog entry I swore I was going to tell you the exact opposite – that a good idea even in the absence of execution is still valid. C’est la vie. If I can’t convince myself of that, I certainly won’t be able to convince you.

Why I don’t like process standardization

I think I’ll end up writing more entries in the first few days of this blog and then they’ll probably taper off.  Why, you ask?  Well, a while ago I started keeping notes in a little Word document on my desktop at work.  Each one was a short thought about something I had observed.  For the moment, it’s that list of “truths” that I’m converting into rants.  I probably should pace myself, but I so enjoyed writing last night’s entry that I wanted to write another today.

The topic is process standardization.  First off let me define this in terms of software development.  For most people this would mean creating a standard methodology by which software is created.  You’re probably familiar with common types of methodologies – waterfall, spiral, agile, etc.  Of course, I have read somewhere (wish I could recall to cite it), that one can look at a process as prescriptive or descriptive.  The difference being that a descriptive process tells your sort of, kind of, perhaps what you should do in a given situation while a prescriptive process tells you “this is the way it is, and there is no other way in which you ought to do it (whatever ‘it’ is).”

I will first say that I’m inclined to believe that descriptive process is useless.  If it doesn’t lay out a specific pattern of behavior to be used over and over again, it is NOT a process.  It’s just a collection of tools you can use to solve a problem.  Like all good artisans, there are individuals who can perform under these circumstances.  Give such a person a block of wood and a box full of tools and he may create for you a beautifully sculpted creature.  However, MOST people don’t fit into the good artist category – most people in my opinion are mediocre in most if not all ways.  Sure, there are things each person may be good at, but being a piano virtuoso has little value when you’re developing software.  So, with that in mind, we create prescriptive processes.  It is these processes that allow mediocre people to use the same set of tools that an artist uses to create a passably sculpted creature by repeating some set of steps.  And, of course, to compensate, you replace free-handing stuff with templates to assure you get consistent results.  Speaking of sculpting, that reminds me that in a future rant I want to talk about the right way to carve an elephant…

OK, so back to my point about process standardization now that we’ve assumed we have a prescriptive process to work with.  At some level, process standardization makes sense to me.  For example, when I managed a group of developers and they had no process at all, I put into place a strict waterfall methodology which included a detailed template for each phase (analysis, design and test).  It also included review checklists for peer reviews and it was all enforced by a workflow management tool so people couldn’t skip steps.  Standardization was good here.  By getting all my developers working the same way, consistently getting their work reviewed by their peers, and assuring that they had not made common mistakes I was able to virtually eliminate regression defects and I cut the average time in QA by half for every release we put out.  A standard process is good… to a point.

Years later I ended up getting Six Sigma training.  One of the key building blocks of Six Sigma is y = f(xi, xp).  Y is a function of xi and xp.  Translated, “the output of your process is a factor of the process you use (xp) and the inputs to that process (xi).”  Not only have I said that standard process is good (to a point), but I also agree with this particular formula.  It’s very intuitive but powerful.  For example, if you ask me to produce metal widgets but my factory creates them whatever way each individual worker feels is right (the process) you can bet that my widgets won’t come out consistent.  Additionally, even if I totally automate my factory to take in raw steel and roll out metal widgets, if I dump plastic (the inputs) into the hopper, I am not going to get good METAL widgets.  What it tells you is that you need both the right inputs (the right requirements, the right developers, the right development tools, etc.) AND a good, consistent way to manipulate the inputs so the output is good.

And that’s where it stops, or at least it should be.  Here’s where I think the over-zealousness of Six Sigma people goes wrong in the software development space.  I’m working on improving the way I develop product X.  You can measure the quality of product X any way you like, let’s say in software defects found after delivery to QA.  That can include all defects found throughout QA and into production.  It doesn’t really matter.  Let’s say that given the set of inputs I have for this product X, I optimize the product until I have very few defects.  This is a good thing.  So, what’s the silly Six Sigma person do?  They attempt to re-use the process on product Y.

NO!  NO!  NO!  The formula isn’t y = f(xp)!  It is y = f(xi,xp), right?  And guess what, by being a different product, product Y naturally has a different set of inputs than product X.  The software of which product Y is comprised is probably written in a different language, by different developers, with different skill levels.  It certainly has different features than product X, or else you wouldn’t have product Y, now would you?  So, why would you use the same process to manipulate a different set of inputs?  I’m never going to make metal widgets out of plastic.  Honestly, it bugs the heck out of me.  Look at each logical grouping and apply a process to that.  Don’t re-use a process just because it worked somewhere else.