Is it an enhancement or a bug?

Did I write this before? I can’t recall, but it came to mind today at work in a conversation I was having. From a user’s perspective, whether you intended the code that way or not, whether it is a “bug” (I didn’t intend it to do that and you don’t like it) or an “enhancement” (I did intend that, but you don’t like it), doesn’t matter.

The distinction we make on this topic is one of vanity. We want to somehow feel better for having done as we were told and yet not satisfied our customer. How is it ok to just be the order taker, to claim that we have done as asked, and that the change requested is an “enhancement”? In effect, “this is not my fault.”

It is your fault. It is OUR fault. We are one company, one organization, with a single customer (or group of customers, it doesn’t really matter) to serve. If we fail to serve that market, whether it is through coding incompetence (bugs) or failure to recognize the need (enhancements) doesn’t matter.

What matters is that we aren’t meeting their need, and beyond that, the distinction we make between bug and enhancement is only a small piece of information in the search for a preventable root cause. This isn’t a hard concept to grasp, yet we seem to struggle with it all the time.

If the customer doesn’t like it, it doesn’t matter the goodness of our intention. It must be remedied. The customer does not care how the undesirable behavior was introduced, just that it exists at all.

100 metrics you mostly shouldn’t bother with

Because I tend to focus my comments on software development processes, I subscribe to a number of newsletters just to keep abreast of stuff that other people are writing. Generally, stuff is a rehash of what we’ve already heard – see my commentary on why it seems nobody is doing primary research anymore. Today I got an interesting one which was “100 IT Performance Metrics”.

I’ve decided to rename it to “100 metrics you mostly shouldn’t bother with.” Don’t get me wrong, I love metrics and data but there are just too many issues with this proposal from Mr. Spanos to ignore.

1. 100 is way too much information. It violates the idea of the critical few things you ought to control. And there are a critical few things which really make most of the difference.
2. Wrong statistic. Mr. Spanos suggests the average in a lot of places where the median would likely be more appropriate. See numbers 3, 4, 9, 10, 11, etc.
3. Rather than reporting one useful metric, he recommends many more than necessary. Boxplots, in many cases would better display the data than 3-4 graphs. For example, numbers 3 and 6 (mean and max resolution time) could be built into a single graph which also showed interquartile range, outliers and providing an easy to read month-over-month comparison. Just doing this could cut the number of proposed metrics in half or more.
4. Stratification hell – by type, by severity. Enough said. If you solve a problem and have fewer incidents the whole number will go down. It is highly unlikely that the resolution of high severity incidents will be replaced in equal volumes by medium severity incidents. Just measure the overall number, once.
5. Lack of scale. All these measures lack any sense of scale. Unlike a factory, the amount of work an IT shop is doing varies greatly from month to month. Without a normalizing factor, increases (or decreases) in any of these measures might entirely be due to changes in work volume.
6. Irrelevance. Average contractor cost? Here’s a place where median is a necessity, since one expensive contractor will blow this number out of the water. Also, who cares? Are you in the business of measuring what rate the market will bear or the rate of inflation year over year? If you’ve chosen to use contractors, you’re going to have to pay for them.
7. Unmeasurable. Change success rate. What the heck is a successful change? One that makes it to production? One that makes it to production with no bugs? One that yields no bugs once in production?
8. Lagging. All these metrics are after-the-fact. If you live by these metrics you have to wait for bad things to happen to you before you realize it. Find some leading indicators of when you’ll be off budget or off schedule or off quality targets and use those instead.

I could go on, but why bother. This is just too easy to pick apart. If you think implementing 100 metrics is the right thing to do, you’re off your rocker. 50 is probably too many. 25 is too. Think critical few things that make your business tick. I’d guess the number is probably 3-4 output metrics and 3-4 input metrics per each output – somewhere in the range of 12 to 16 measures should get you most of the way there.

Just the conclusions?

I happened to be cleaning out my home office for some reason when I stumbled upon one of my old performance reviews. Given how dreadfully most companies conduct performance reviews, I usually pay little heed to the words written and just look at the dollars. It becomes a “putting your money where your mouth is” thing. Either the dollars match what you are saying or they don’t. If you say nice things but give a mediocre raise, then you can be sure that the words are hollow.

At any rate, I was (despite my tendency to just look at the dollars) re-reading the comments on this particular review. One of the comments was “… needs to tailor his presentation to the level of his audience. Senior individuals only want the conclusions… “

I have been well aware of the idea of tailoring to your audience for a long, long time. I think it’s one of those first thing you learn. Surely the most senior individual wants nothing to do with the dull details of your code, but what about the details of ones analysis? True, they probably don’t care about homoscedasticity in your data, but just the conclusions? That seems like the other end of the extreme – too little information. Is this really the right thing to do?

I think, if you ponder this proposal for a minute you will realize it’d be a foolish person who’d just come in with the conclusions. If I concluded to anyone, at any level, that “the world is flat” then I suspect the reasonable question they’d ask is “why do you think that is so?” This isn’t a question of tailoring to one’s level appropriately. No matter who you are, you can’t walk into a presentation with just the conclusions.

Why not? Any idiot (or liar) can write down a bunch of “conclusions.” In the absence of the data which shows why you believe what you believe, a slide of conclusions is worthless. Yes, there are nitty-gritty details of doing analysis, like how I transposed these columns of data into that pivot table in order to run my Mood’s Median test, which should be dropped from the presentation. But that’s the type of thing I wouldn’t explain to even the most detail oriented of my peers. Those details are irrelevant to everyone.

The issue isn’t what you should present to a senior individual, but whether what you have is worthy of any attention at all. Senior managers are still people whose brains work just like the brains of their subordinates. They need information to make decisions just as we all do, but they need information about different types of problems. If they don’t need to make a decision, what are you doing there at all presenting? And most importantly, if they asked for the information, then you can reasonably assume they wanted to hear more than the conclusion.

Bring along enough information to tell a complete story; far more than just the conclusions. If they just wanted the conclusion, you could send an email instead.

Uneventful change

As I was sitting here doing nothing, or effectively nothing (I was playing mini-games on my computer), for some reason a bunch of failed change events came into my head. As my prior post would indicate, I fail to make change a lot of the time, though not for a lack of trying.

To be fair, if I make a change and it isn’t better, then I hardly count it as making a change. Sure, something is different but not in a way that is meaningful to our customer.

Anyway, I digress. I was thinking about change events and why they always seem to be such a big event. It brought to mind a story about trying to build a new requirements process. Years ago, I worked for a team who was doing a process redesign for their requirements process. It was going to be this big multi-day event where we brought everyone offsite to hash out a new way to do requirements.

Never mind the fact that I believe this type of work really needs to be done scientifically as opposed to just throwing people into a room and saying “design something.” Anyway, I was tasked to go off and get a resource from this one team to attend. I approached the vice president of the team and said something along the lines of “I’d like to have a representative from your team attend this 3 day offsite to redesign the requirements process.”

“No,” he replied. I sat there and stared at him. I really didn’t know what to say. It wasn’t an angry stare on my part, it was more like a mouth agape bewildered stare. Kind of like the way my dog cocks his head to the side when something makes a funny noise. I’m pretty sure that’s what I looked like. Fortunately, he went on.

“We already redesigned this process once with Bob [a
coworker of mine]. Why should I send resources to do it again?”

Fair enough. It was true; Bob had tried to do this process redesign before. It had failed not because they were unable to come to an agreement over the change but because the team who was supposed to adopt the change refused. Foolishly, Bob had neglected a major stakeholder in the process. But, darn it all, we weren’t going to make that mistake again.

And yet we did. The mistake wasn’t trying to get all the people in the room, or doing it via an offsite, or doing it without understanding the existing process fully. The mistake was making change an event. An event is easy to resist. And event has a clear beginning and thus a clear line in the sand you can take a stand against.

What if change just snuck up on you? It happens all the time in our regular lives and we handle them in stride pretty well. Tools like twitter, Facebook, even the internet and email just creep into our lives without making a big scene. It’s hard to resist something that shows up quietly, that doesn’t really come knocking or announcing itself. It just shows up because it is better and more useful. Now that we have these things, we can’t imagine life without them.

Why can’t we approach process change this way? What if instead of a big-bang event to redesign the requirements process we just made some small quiet change to the way we did things? What if we just started inviting teams to JAR sessions. No training, no announcements that this was the new way to do things. Just invite them on some projects and see how they feel about it. If they like it, maybe we do it on some more projects. Eventually we’re doing it all the time and it becomes an accepted way of doing things.

There’s no real need, except potentially for speed purposes, to announce a process change. And given the enormous resistance that can be placed against an event, perhaps it is just better to sneak a little change for the better here or there. Do it often and eventually the process will be changed for the better, and significantly so.

Perhaps a better title for this posting would be “eventless change,” but the idea is the same. Change doesn’t have to be eventful for it to be impactful.

I fail… A LOT

Failure may well be the hallmark of Six Sigma work everywhere. I know what you’re thinking… “that’s exactly why people shouldn’t use Six Sigma! It doesn’t work!” But, in fact, I am here to argue that how often you fail may be a measure of how well Six Sigma is working.

If you’re in software development you know that there are no silver bullets for the problems that appear to plague us. (If you believe otherwise, well, I’m really not sure what to tell you) Every technology trend to date has yet to generate any major change in productivity or quality for a disproportionately low investment. Third generation languages, fourth generation languages, APIs, automation, static code analysis, code reuse, Agile… none of them have been the miracle they claim to be. They in essence, failed to live up to their expectations.

But a few proven good things have come out. They’re not silver bullets, but they’re very powerful when used properly… use cases and Fagan-style inspections come to mind. Most of what has come out though… failures. CASE… bleh. Code generators… bleh. Requirements management solutions… ho hum. Simply put, in a free market anything that would give such a disproportionate advantage to your competition would be quickly adopted by your own team. That hasn’t universally happened for almost anything I can think of.

And as I looked back on all the times I’ve tried to make process change and how many times it hasn’t worked, I can see how someone would say “Six Sigma = failure”. It’s a bad overgeneralization. Six Sigma often has failures, but it also has wins. And if you look at the wins, they’re often disproportionate wins – little effort, major breakthrough. Not always does this happen. Rarely, in fact. But when it does, Six Sigma earns its keep.

The nature of my work is such that I often will find almost no benefit, or very little, from the process change I tried. It seems like a good idea but it just doesn’t work out or it is hard to maintain or it actually makes things worse. But every now and again I hit one out of the park.

The reality of process improvement is that most of the times at bat (to continue the metaphor) you strike out. Accept that you will fail a lot when trying to make something better. And then, once in a rare while, you’ll have a huge success – a grand slam, as it were.

Failures and Six Sigma do go hand in hand. But failures are a measurement of how many times you tried. You can’t fail if you don’t try, but you can’t succeed either. Fail often because that means trying often.

Making someone else do it isn’t lean

Being lean is about the elimination of non-value added work, but this consideration should not be made in a vacuum. At the end of an otherwise successful (a bit of a rarity) meeting the other day, I was talking with the leaders of a couple parts of the quality assurance organization. They were telling me how they were getting leaner and not doing testing when it wasn’t necessary.

I asked them how that was possible. They said “we’ve asked the developer to decide whether or not testing is appropriate” and went on to explain the various elements of the decision tree the developer would have to go through. I said, “why don’t you just make that decision? You’re just as capable of determining it.”

“But then we’d have to do that work, and that wouldn’t be lean.”

Hello? Forcing work out of your team and on to someone else’s team is not leaning out the process. It is just pushing it around. Consider the auto industry; many of them claim to be leaner for having forced their suppliers to do most of the assembly work for them. Yes, it is true that the maker of your car didn’t do the work anymore, so they didn’t have to pay the labor or spend the time doing the assembly, but that didn’t mean the work went away. It just went somewhere else.

It’s a reminder that we have to look at a process holistically to truly lean it out. It isn’t a question of “do I have to do this thing?” but of “do any of us have to do this thing?” A waste is a waste no matter who is responsible for creating it. If you ask the first question, you do what the QA team did and just pushed the waste around. If you ask the second question instead, maybe you just decide you don’t need to do the thing at all.

I’ll leave you with a good counter-example from a software textbook I was reading about designing for the users instead of just solving the problem you could with the computer (sorry, I can’t remember the name of the text at the moment). Essentially what was happening was that the post office was delivering renewals, new memberships and cancellations all to this company. The company wanted to solve the labor problem with software, but ended up making it worse. One of the things that was problematic was that each type of request (renewal, new member, cancel) needed a different process flow, but that meant all incoming mail had to be opened and then sorted by the request type before it could be processed. That’s just pure waste (motion, transportation, or over-processing depending on how exactly it was done). The solution to this un-lean thing was to push the problem to the postal service. No, the postal service didn’t start opening and sorting the mail by the contents. Instead, they assigned each type of correspondence a different PO box, so when you stuffed your request into the pre-labeled envelope the post office did what it does best and automatically sorted it into the right bin.

Sure, the occasional piece was in the wrong place, but by leveraging the post office’s existing capability to examine and sort the mail (which they were going to do anyway) you eliminated the NVA step of having the re-sort it at the destination. A simple change, no more work for anyone involved. That’s leaner!

Making someone else do your work for you, that’s just a way to make them mad.

Estimation from history

If we don’t study the past we are doomed to repeat it, any historian (or history buff) will tell you. And yet, when it comes to estimating, if we DO study the past then we are doomed to repeat it.

Of all the potential methods of doing estimation, I thought that building up knowledge about past performances would be one of the most useful for having future estimates which are accurate. Ideally, as enough time passes you begin to see repeating patterns in your projects and can leverage that knowledge to assemble new estimates that are based on the actual prior experiences.

In a journey of trying to continuously improve, what could be worse? Using your historical performances to predict the future is valuable if you are happy with your past performance. The market is never happy with your past performance. Even if that performance is good, your competition will be looking to best you. If you build estimates from your past then you are going to be repeating your past. Parkinson’s law will take care of that for you, even if you try to become more productive.

It may seem like a subtle tweak to using a historical database to create estimates, but I think it is an important one… you can’t just estimate based on past performance. You can use that knowledge, but then you have to be prepared to set a goal for your project that is 10% (or some other percent) better than the last time you did it. You must always be striving for better. Whether you achieve it or not, the culture must be that no matter how good what you have done in the past is, we must always be looking for more. Process improvement is not an event, it is a journey.