The difference between knowing and thinking you know

I was talking with one of my managers the other day when she presented an interesting point that I thought I’d share. There is a difference between knowing something and thinking you know something. Sure, there is pure ignorance, but it’s not the type of issue we were talking about.

The type of intellectual leap you shouldn’t make is the taking of incomplete knowledge and representing it as complete knowledge. It’s the overextension of a specific situation to being the general problem. For example, we know that some defects were introduced when the code was written years ago and are just appearing now, but that doesn’t mean that our problem is all latent bugs. This is inductive fallacy. All we’ve seen (for those that we can confirm at all) is that some bugs exist since the creation of the code, but this does not make the general statement “our issue is latent bugs” necessarily true.

The list of bad extensions is long, and yet easy to remedy. We should be talking about odds and probabilities and uncertainty and recognizing that it exists with much of what we deal with. For example, Stephen Kan, a researcher from IBM, recognized that test execution doesn’t happen linearly but instead follows and S-shaped curve. When we explored our patterns of test execution we discovered the same thing held true of us. And so, knowing that, we could set forth a realistic pattern of what would have to happen (for a given set of tests and duration to complete the testing) in order for us to be done one time. We tried it out, and found out that we don’t always perfectly follow the curve.

It wasn’t about perfectly following the curve. It was about more-or-less following it. There are essentially bands, plus or minus, around this idealized curve which represent the uncertain area which is normal variation from the perfect shape. As long as you stay within those, we feel comfortable that things are on track. But we represent the uncertainty, not ignore it, because it is there. The knowledge says “testing follows and s-shape”, but we don’t make that an absolute truth, just a general pattern that includes room for representing that we don’t understand everything that may cause it to deviate from that path.

"A little learning is a dangerous thing; drink deep, or taste not the Pierian spring: there shallow draughts intoxicate the brain, and drinking largely sobers us again."

- Alexander Pope, An Essay on Criticism

His quote is not just a warning that knowing a little causes you to do bad things, but that it is necessary for us drink deeply, acknowledge that we do not know everything and seek to figure out what it is we do not know.

Is it an enhancement or a bug?

Did I write this before? I can’t recall, but it came to mind today at work in a conversation I was having. From a user’s perspective, whether you intended the code that way or not, whether it is a “bug” (I didn’t intend it to do that and you don’t like it) or an “enhancement” (I did intend that, but you don’t like it), doesn’t matter.

The distinction we make on this topic is one of vanity. We want to somehow feel better for having done as we were told and yet not satisfied our customer. How is it ok to just be the order taker, to claim that we have done as asked, and that the change requested is an “enhancement”? In effect, “this is not my fault.”

It is your fault. It is OUR fault. We are one company, one organization, with a single customer (or group of customers, it doesn’t really matter) to serve. If we fail to serve that market, whether it is through coding incompetence (bugs) or failure to recognize the need (enhancements) doesn’t matter.

What matters is that we aren’t meeting their need, and beyond that, the distinction we make between bug and enhancement is only a small piece of information in the search for a preventable root cause. This isn’t a hard concept to grasp, yet we seem to struggle with it all the time.

If the customer doesn’t like it, it doesn’t matter the goodness of our intention. It must be remedied. The customer does not care how the undesirable behavior was introduced, just that it exists at all.

100 metrics you mostly shouldn’t bother with

Because I tend to focus my comments on software development processes, I subscribe to a number of newsletters just to keep abreast of stuff that other people are writing. Generally, stuff is a rehash of what we’ve already heard – see my commentary on why it seems nobody is doing primary research anymore. Today I got an interesting one which was “100 IT Performance Metrics”.

I’ve decided to rename it to “100 metrics you mostly shouldn’t bother with.” Don’t get me wrong, I love metrics and data but there are just too many issues with this proposal from Mr. Spanos to ignore.

1. 100 is way too much information. It violates the idea of the critical few things you ought to control. And there are a critical few things which really make most of the difference.
2. Wrong statistic. Mr. Spanos suggests the average in a lot of places where the median would likely be more appropriate. See numbers 3, 4, 9, 10, 11, etc.
3. Rather than reporting one useful metric, he recommends many more than necessary. Boxplots, in many cases would better display the data than 3-4 graphs. For example, numbers 3 and 6 (mean and max resolution time) could be built into a single graph which also showed interquartile range, outliers and providing an easy to read month-over-month comparison. Just doing this could cut the number of proposed metrics in half or more.
4. Stratification hell – by type, by severity. Enough said. If you solve a problem and have fewer incidents the whole number will go down. It is highly unlikely that the resolution of high severity incidents will be replaced in equal volumes by medium severity incidents. Just measure the overall number, once.
5. Lack of scale. All these measures lack any sense of scale. Unlike a factory, the amount of work an IT shop is doing varies greatly from month to month. Without a normalizing factor, increases (or decreases) in any of these measures might entirely be due to changes in work volume.
6. Irrelevance. Average contractor cost? Here’s a place where median is a necessity, since one expensive contractor will blow this number out of the water. Also, who cares? Are you in the business of measuring what rate the market will bear or the rate of inflation year over year? If you’ve chosen to use contractors, you’re going to have to pay for them.
7. Unmeasurable. Change success rate. What the heck is a successful change? One that makes it to production? One that makes it to production with no bugs? One that yields no bugs once in production?
8. Lagging. All these metrics are after-the-fact. If you live by these metrics you have to wait for bad things to happen to you before you realize it. Find some leading indicators of when you’ll be off budget or off schedule or off quality targets and use those instead.

I could go on, but why bother. This is just too easy to pick apart. If you think implementing 100 metrics is the right thing to do, you’re off your rocker. 50 is probably too many. 25 is too. Think critical few things that make your business tick. I’d guess the number is probably 3-4 output metrics and 3-4 input metrics per each output – somewhere in the range of 12 to 16 measures should get you most of the way there.

Just the conclusions?

I happened to be cleaning out my home office for some reason when I stumbled upon one of my old performance reviews. Given how dreadfully most companies conduct performance reviews, I usually pay little heed to the words written and just look at the dollars. It becomes a “putting your money where your mouth is” thing. Either the dollars match what you are saying or they don’t. If you say nice things but give a mediocre raise, then you can be sure that the words are hollow.

At any rate, I was (despite my tendency to just look at the dollars) re-reading the comments on this particular review. One of the comments was “… needs to tailor his presentation to the level of his audience. Senior individuals only want the conclusions… “

I have been well aware of the idea of tailoring to your audience for a long, long time. I think it’s one of those first thing you learn. Surely the most senior individual wants nothing to do with the dull details of your code, but what about the details of ones analysis? True, they probably don’t care about homoscedasticity in your data, but just the conclusions? That seems like the other end of the extreme – too little information. Is this really the right thing to do?

I think, if you ponder this proposal for a minute you will realize it’d be a foolish person who’d just come in with the conclusions. If I concluded to anyone, at any level, that “the world is flat” then I suspect the reasonable question they’d ask is “why do you think that is so?” This isn’t a question of tailoring to one’s level appropriately. No matter who you are, you can’t walk into a presentation with just the conclusions.

Why not? Any idiot (or liar) can write down a bunch of “conclusions.” In the absence of the data which shows why you believe what you believe, a slide of conclusions is worthless. Yes, there are nitty-gritty details of doing analysis, like how I transposed these columns of data into that pivot table in order to run my Mood’s Median test, which should be dropped from the presentation. But that’s the type of thing I wouldn’t explain to even the most detail oriented of my peers. Those details are irrelevant to everyone.

The issue isn’t what you should present to a senior individual, but whether what you have is worthy of any attention at all. Senior managers are still people whose brains work just like the brains of their subordinates. They need information to make decisions just as we all do, but they need information about different types of problems. If they don’t need to make a decision, what are you doing there at all presenting? And most importantly, if they asked for the information, then you can reasonably assume they wanted to hear more than the conclusion.

Bring along enough information to tell a complete story; far more than just the conclusions. If they just wanted the conclusion, you could send an email instead.

Uneventful change

As I was sitting here doing nothing, or effectively nothing (I was playing mini-games on my computer), for some reason a bunch of failed change events came into my head. As my prior post would indicate, I fail to make change a lot of the time, though not for a lack of trying.

To be fair, if I make a change and it isn’t better, then I hardly count it as making a change. Sure, something is different but not in a way that is meaningful to our customer.

Anyway, I digress. I was thinking about change events and why they always seem to be such a big event. It brought to mind a story about trying to build a new requirements process. Years ago, I worked for a team who was doing a process redesign for their requirements process. It was going to be this big multi-day event where we brought everyone offsite to hash out a new way to do requirements.

Never mind the fact that I believe this type of work really needs to be done scientifically as opposed to just throwing people into a room and saying “design something.” Anyway, I was tasked to go off and get a resource from this one team to attend. I approached the vice president of the team and said something along the lines of “I’d like to have a representative from your team attend this 3 day offsite to redesign the requirements process.”

“No,” he replied. I sat there and stared at him. I really didn’t know what to say. It wasn’t an angry stare on my part, it was more like a mouth agape bewildered stare. Kind of like the way my dog cocks his head to the side when something makes a funny noise. I’m pretty sure that’s what I looked like. Fortunately, he went on.

“We already redesigned this process once with Bob [a
coworker of mine]. Why should I send resources to do it again?”

Fair enough. It was true; Bob had tried to do this process redesign before. It had failed not because they were unable to come to an agreement over the change but because the team who was supposed to adopt the change refused. Foolishly, Bob had neglected a major stakeholder in the process. But, darn it all, we weren’t going to make that mistake again.

And yet we did. The mistake wasn’t trying to get all the people in the room, or doing it via an offsite, or doing it without understanding the existing process fully. The mistake was making change an event. An event is easy to resist. And event has a clear beginning and thus a clear line in the sand you can take a stand against.

What if change just snuck up on you? It happens all the time in our regular lives and we handle them in stride pretty well. Tools like twitter, Facebook, even the internet and email just creep into our lives without making a big scene. It’s hard to resist something that shows up quietly, that doesn’t really come knocking or announcing itself. It just shows up because it is better and more useful. Now that we have these things, we can’t imagine life without them.

Why can’t we approach process change this way? What if instead of a big-bang event to redesign the requirements process we just made some small quiet change to the way we did things? What if we just started inviting teams to JAR sessions. No training, no announcements that this was the new way to do things. Just invite them on some projects and see how they feel about it. If they like it, maybe we do it on some more projects. Eventually we’re doing it all the time and it becomes an accepted way of doing things.

There’s no real need, except potentially for speed purposes, to announce a process change. And given the enormous resistance that can be placed against an event, perhaps it is just better to sneak a little change for the better here or there. Do it often and eventually the process will be changed for the better, and significantly so.

Perhaps a better title for this posting would be “eventless change,” but the idea is the same. Change doesn’t have to be eventful for it to be impactful.

I fail… A LOT

Failure may well be the hallmark of Six Sigma work everywhere. I know what you’re thinking… “that’s exactly why people shouldn’t use Six Sigma! It doesn’t work!” But, in fact, I am here to argue that how often you fail may be a measure of how well Six Sigma is working.

If you’re in software development you know that there are no silver bullets for the problems that appear to plague us. (If you believe otherwise, well, I’m really not sure what to tell you) Every technology trend to date has yet to generate any major change in productivity or quality for a disproportionately low investment. Third generation languages, fourth generation languages, APIs, automation, static code analysis, code reuse, Agile… none of them have been the miracle they claim to be. They in essence, failed to live up to their expectations.

But a few proven good things have come out. They’re not silver bullets, but they’re very powerful when used properly… use cases and Fagan-style inspections come to mind. Most of what has come out though… failures. CASE… bleh. Code generators… bleh. Requirements management solutions… ho hum. Simply put, in a free market anything that would give such a disproportionate advantage to your competition would be quickly adopted by your own team. That hasn’t universally happened for almost anything I can think of.

And as I looked back on all the times I’ve tried to make process change and how many times it hasn’t worked, I can see how someone would say “Six Sigma = failure”. It’s a bad overgeneralization. Six Sigma often has failures, but it also has wins. And if you look at the wins, they’re often disproportionate wins – little effort, major breakthrough. Not always does this happen. Rarely, in fact. But when it does, Six Sigma earns its keep.

The nature of my work is such that I often will find almost no benefit, or very little, from the process change I tried. It seems like a good idea but it just doesn’t work out or it is hard to maintain or it actually makes things worse. But every now and again I hit one out of the park.

The reality of process improvement is that most of the times at bat (to continue the metaphor) you strike out. Accept that you will fail a lot when trying to make something better. And then, once in a rare while, you’ll have a huge success – a grand slam, as it were.

Failures and Six Sigma do go hand in hand. But failures are a measurement of how many times you tried. You can’t fail if you don’t try, but you can’t succeed either. Fail often because that means trying often.

Making someone else do it isn’t lean

Being lean is about the elimination of non-value added work, but this consideration should not be made in a vacuum. At the end of an otherwise successful (a bit of a rarity) meeting the other day, I was talking with the leaders of a couple parts of the quality assurance organization. They were telling me how they were getting leaner and not doing testing when it wasn’t necessary.

I asked them how that was possible. They said “we’ve asked the developer to decide whether or not testing is appropriate” and went on to explain the various elements of the decision tree the developer would have to go through. I said, “why don’t you just make that decision? You’re just as capable of determining it.”

“But then we’d have to do that work, and that wouldn’t be lean.”

Hello? Forcing work out of your team and on to someone else’s team is not leaning out the process. It is just pushing it around. Consider the auto industry; many of them claim to be leaner for having forced their suppliers to do most of the assembly work for them. Yes, it is true that the maker of your car didn’t do the work anymore, so they didn’t have to pay the labor or spend the time doing the assembly, but that didn’t mean the work went away. It just went somewhere else.

It’s a reminder that we have to look at a process holistically to truly lean it out. It isn’t a question of “do I have to do this thing?” but of “do any of us have to do this thing?” A waste is a waste no matter who is responsible for creating it. If you ask the first question, you do what the QA team did and just pushed the waste around. If you ask the second question instead, maybe you just decide you don’t need to do the thing at all.

I’ll leave you with a good counter-example from a software textbook I was reading about designing for the users instead of just solving the problem you could with the computer (sorry, I can’t remember the name of the text at the moment). Essentially what was happening was that the post office was delivering renewals, new memberships and cancellations all to this company. The company wanted to solve the labor problem with software, but ended up making it worse. One of the things that was problematic was that each type of request (renewal, new member, cancel) needed a different process flow, but that meant all incoming mail had to be opened and then sorted by the request type before it could be processed. That’s just pure waste (motion, transportation, or over-processing depending on how exactly it was done). The solution to this un-lean thing was to push the problem to the postal service. No, the postal service didn’t start opening and sorting the mail by the contents. Instead, they assigned each type of correspondence a different PO box, so when you stuffed your request into the pre-labeled envelope the post office did what it does best and automatically sorted it into the right bin.

Sure, the occasional piece was in the wrong place, but by leveraging the post office’s existing capability to examine and sort the mail (which they were going to do anyway) you eliminated the NVA step of having the re-sort it at the destination. A simple change, no more work for anyone involved. That’s leaner!

Making someone else do your work for you, that’s just a way to make them mad.

Estimation from history

If we don’t study the past we are doomed to repeat it, any historian (or history buff) will tell you. And yet, when it comes to estimating, if we DO study the past then we are doomed to repeat it.

Of all the potential methods of doing estimation, I thought that building up knowledge about past performances would be one of the most useful for having future estimates which are accurate. Ideally, as enough time passes you begin to see repeating patterns in your projects and can leverage that knowledge to assemble new estimates that are based on the actual prior experiences.

In a journey of trying to continuously improve, what could be worse? Using your historical performances to predict the future is valuable if you are happy with your past performance. The market is never happy with your past performance. Even if that performance is good, your competition will be looking to best you. If you build estimates from your past then you are going to be repeating your past. Parkinson’s law will take care of that for you, even if you try to become more productive.

It may seem like a subtle tweak to using a historical database to create estimates, but I think it is an important one… you can’t just estimate based on past performance. You can use that knowledge, but then you have to be prepared to set a goal for your project that is 10% (or some other percent) better than the last time you did it. You must always be striving for better. Whether you achieve it or not, the culture must be that no matter how good what you have done in the past is, we must always be looking for more. Process improvement is not an event, it is a journey.

Return on Investment

I’m sure there are a fair number of places which cover what exactly a return on investment is, so I’m going to keep that part of my post brief. In short, a return on investment (ROI, hereafter) is a calculation of how long it will take you to make your money back.

For example, let’s say you spend $100 to tune up your car. Each month thereafter, you use $10 less in gasoline because your car runs more efficiently. Therefore, your ROI is $100 / $10 = 10 months until you make your money back. There are some subtleties that businesses add to that. Typically you discount future dollars saved because a dollar in the future is worth less than a dollar now.

Ultimately, the goal of doing an ROI is to decide whether to invest or not. Businesses typically look to make their money back in 3 to 5 years on an investment. The best ROIs are selected to decide where to spend your money first among many competing initiatives which want budget money. Just like in your personal life, businesses have to make decisions about what to spend your limited funds on. One alternative to any investment is always to sock the money away in some investment and just grow it. There’s no obligation to spend the money since it must always be weighed against whether it’d just be worth it to invest the cash instead.

Anyway, I digress. Having said all that, the way most businesses present an ROI is as some set of expenses divided by some amount of savings. People brainstorm what the expense factors are (labor, materials, etc.) in order to complete the project and then estimate the benefits. I don’t have any issue with the basic math that people are doing for an ROI. For the most part, people can handle the addition, subtraction, multiplication and division necessary.

Where people go wrong on an ROI is filling in the variables. Let’s take a simple example. I believe that if I create comprehensive system documentation that I can save time in design and development of new code because people won’t have to read the code to figure out how to integrate their new features. It’s pretty logical sounding, right? So, let’s say I estimate it’ll cost $100,000 to create the documentation and I estimate that it’ll save $25,000 in time…. Wait. What?

Where did my numbers come from? I made them up. And that’s what most people are doing when they do an ROI. They estimate some costs (which we tend to be more capable at doing) and then we make up some savings we’ll realize to make the ROI work. Rather than saying “well, based on past performance data projects cost us between X and Y and based on industry research we’ll probably realize between A and B savings” people just make up the numbers to make the ROI work. We already know what the goal is – an ROI in 3 to 5 years.

The problem shifts from an honest analysis of what the costs and benefits are to what numbers I need to plug in to get into that 3 to 5 year range. Microsoft Excel has a handy “goal seek” function you can even use for this. Don’t do it!

Everyone wants their idea to be invested in. Everyone can handle basic math (more or less). The problem with ROIs is all in the assumptions people make about the value that a certain variable will take on. Those values should be based on some sort of real data whether it’s neutral industry research (don’t EVER trust a company selling something), expert opinions (you should always get several opinions and account for the uncertainty), or researched data. Furthermore, you should ALWAYS be modeling the fact that results are rarely exactly always the same. Savings on a per experience basis (per project you run, for example) will vary. Many projects are not an automatic home run – there’s a good chance you’ll lose money on the deal – and you have to be honest about the possibility.

We shouldn’t be doing an ROI because we’re trying just to get past the annoyance of an investment committee. We should be doing an ROI to provide an informed view of the opportunity. We all have good ideas, but that doesn’t mean they’re the BEST idea that the company should invest in.

Beware the anecdote

“This one company used our software and they realized a $10 million savings in one year!”

Good for them. Frankly, I don’t (and you shouldn’t) care. The anecdote may be the most disastrous (from the consumer’s perspective, not the sellers) marketing device ever employed – short of outright lying I suppose. The anecdote takes a great story and uses it as the basis for why you should take your hard earned money and buy product X, or Y or Z.

Anecdotes abound. We are a culture of story tellers. The media exploits it all the time. The government exploits it all the time. Just think about the current health care debate. On one side you have the story of the Canadian woman who would have died had she relied on Canada’s healthcare. On the other side, you have a very happy Canadian health care recipient. How can it be that two totally different anecdotes are coming out of the very same system?

It’s a logical fault that we make. We assume that the related experience is the representative experience, which it clearly isn’t. Well, at least not all the time. Consider the “bad” Canadian health care experience. Let’s assume, first of all, that she really would have died if she hadn’t come to the US. We don’t actually know that because we can’t explore alternate realities. She believes she would have died, so let’s take it at face value. Has anyone in the US health care system ever died due to a bad diagnosis? If you answered “no”, I’d like to have some of whatever drug you are on, because that stuff is creating the kind of delusions not seen since the Emperor went out without his clothes on.

Of course there have been bad US experiences! People die here too! But the anecdote is powerful. It’s scary that someone might slip through the cracks. And statistics are oh so dreadfully boring! If Wikipedia is to be believed, the US pays twice as much and yet lags other wealthy nations in life expectancy and infant mortality rates. That doesn’t mean people don’t die in these other countries of horrific, potentially preventable deaths in these other countries – they do. But the odds are better. And even with improved odds of you surviving infancy and living longer, there are scary stories to tell.

People relate an anecdote when it supports their view of the world, whatever that may be. However, they’re doing that because the general patterns of the product, process or service aren’t necessarily an experience which causes you to take action.

I mean, would you be all that excited if I said “if the US adopted a health care system like Canada’s it’d probably be about the same care as you get today, but maybe a little cheaper.” Wow! I mean, wow! If that isn’t a call to action, I don’t know what is!?! People are selling things all the time – whether it is software, a change in the healthcare system, or NOT changing the health care system. If you take that anecdote and relate it to another person, who passes it along to another, and so on then it has served its powerful but potentially destructive purpose. Nobody can remember the statistics, but we all remember a good story.

The next time someone tells you a story about why you should take action, instead of listening intently, ask for the data. Then go verify the data. But most of all, beware the anecdote!