I’ve decided that I hate RACIs. There’s just too much wrong with them to make them worth doing. A RACI, in case you are unfamiliar (or RASCI sometimes) is a table which lists tasks along one axis, people (or job roles) along another axis and at the intersection marks someone one or more of R,A,S,C,I (responsible, accountable, supporting, consulted, informed).

Now, depending on the school of thought you subscribe to, the S option may or may not exist. A simple RACI might look something like:

One can thus determine what my day is like. In the morning, we are all responsible for waking up, while if someone doesn’t get up on time it’s Daddy’s head who is going to roll. Then, we’re all responsible for getting dressed, but since Daddy has no fashion sense, it’s Mommy who is accountable for this task. I’m simply informed about our daughter going to school since I’m going to work. Technically, though my daughter is too young to drive herself, she’s the one responsible for going to school while Mommy is the one who gets in trouble if my 3 year old doesn’t show up. I’m responsible for asking about her school day, Mommy is simply informed. She overhears my daughter say (who is only 3 but clearly has the kid role figure out) in response to “what’d you do at school today?” almost always replies “nothing.” Obviously, my daughter being the expert on her school day is consulted on this task. One might also choose Supportive I suppose since I can’t complete the task without her. Dinner is strictly a Daddy job with Mommy having some say about what I make for dinner and the little one just eating what is presented. If we consulted her every night would be chicken nuggets, pizza or noodles. And finally for dessert, the little one is responsible for the eating while Daddy is simply informed. Mommy has the go/no-go decision on this one.

Phew. Now that is one heck of a lot of data all from that little table, and this is a joke of a RACI. No real job that I do is so basic as to have just 3 players and 6 tasks to it. You can easily see that many more tasks might be appropriate even in my daily plan – including showers, dishes, bedtime for my daughter, etc. We actually follow a pretty predictable pattern each day – a process you might say – and yet there’s enormous detail that could go into it. The fact of the matter is that any realistic RACI would get unmanageable and quick.

The second issue I have with RACIs is that they’re a punt for figuring out a good process. If you have to go to all this trouble to figure out who does what in the process, isn’t it a reasonable question to ask if you might have a really un-lean process that simply needs fewer actors or individuals who actually have the authority to both be responsible and accountable for the task to get done? The mere fact that R (the responsible person who does the work) can be separated from the A (the accountable person who gets in trouble if the work doesn’t get done) is crazy! The idea that we’d consult all kinds of people suggests that we don’t have enough knowledge (or the wherewithal to gather it) about the job we’re tasked with doing. And informing? Look, if the person isn’t going to act on the data you are provided (ie, give feedback, which makes the consulted not informed) then why are you giving them the information!?!?

Third, nobody gets these things right anyway. Can more than one person be accountable? Responsible? I think not. And yet, they’re always a mishmash of several people being responsible for the same task. Someone has to be running the show and the moment you allow everyone to be accountable then nobody is accountable and the same goes for responsibility. It just becomes a giant circle of finger-pointing when the task doesn’t get done.

Finally, these things are usually more of a suggestion than a reality. Putting that much effort into something that isn’t going to be followed is insane. Even in my own house, with the six silly tasks that I have, if I’m working late then Mommy becomes responsible for all the tasks that I would normally be. Life marches on, regardless of what my RACI says ought to happen.

Don’t do RACIs. Do a nice swimlane process flow and leave it at that.

Recently I was presented with an ROI for some project.  It included some costs, which weren’t that interesting and it included one hard benefit which was a time savings.  And then it included a series of soft benefits including improved quality, improved customer satisfaction, etc.

Now, as far as I understand it, a hard benefit is one which you can associate a dollar value with and a soft benefit is a benefit which has no dollar value.  But looking above, these benefits aren’t soft benefits.  They’re (potentially) hard benefits.  Quality is worth something since it costs money to fix defects.  Customer satisfaction is worth something because unhappy customers don’t spend their money with you.

These aren’t soft benefits at all.  They’re hard benefits that you didn’t bother to figure out!  But don’t confuse them.  Just because you haven’t figured out the value is very different from having no value.  I’ve run into just one benefit of a process change which was truly a soft benefit.

It was the benefit of feeling more comfortable about what we were delivering.  For example, doing regression testing and finding no bugs has no hard benefit.  Peace of mind that the code is correct is a soft benefit.  Your customer doesn’t care how well you sleep at night and you didn’t prevent any bugs so you had no impact on quality or customer satisfaction.  That peace of mind is a soft benefit.  Everything else is a hard benefit that you don’t know the value of.  If you’re going to mention it in an ROI, figure out what it is worth.  If the change is so small you can’t measure it, then don’t list it as a benefit.  But lets not pretend here you are getting a “soft” benefit when you are truly getting is a minuscule hard benefit.

I was sitting through a meeting the other day and we were talking about the new test case management system that we were installing. It’s nothing special, but one of the features it offers is the ability to track defects along with the test cases. The thing is, as a company we already have 3 other defect tracking systems from different vendors, none of which are integrated with our test case management system. True, it is an annoyance and certainly adds some cost to quality assurance to record the defect when you have to go into a different system.

However, the issue I have is that quality assurance designed the workflow and fields that would go into the defect tracking system. Quality assurance is a (sort-of) user of the tool. They have to enter data into it, so making it convenient for them to enter data is a good thing. However, they aren’t the primary consumer of the data – development is. The reason we enter bugs into the defect tracking system is (besides metrics) so that development gets a detailed account of what went wrong so they can fix it. Although we supply defects into the process, it is development that has to consume our input and use it.

So why then would you have QA define the input for the team who needs the data? It is illogical that QA should determine what information and in what format they’ll be giving it to development. Development, who needs adequate input to fix the bug, should be defining what inputs they need. QA could input more data than development needs if they felt they needed some data recorded too, provided it isn’t interfering with what development needs to get their job done.

Now, maybe development requests something that is ridiculous and QA ought to negotiate for a more acceptable request, but generally I believe that it is the consumer who gets to define what they need from their supplier. In our case, we didn’t ask development at all what they needed, we just built something. We didn’t ask development if they’d be willing to use our new defect tracking system.

I think, if it hasn’t been set out as a universal truth somewhere else before, that it ought to be done so now – the consumer defines the required input, not the supplier.

This evening, my daughter, in an attempt to be helpful offered to put our dog’s food into his bowl for dinner. She does this every now and again, and usually the mess is kept to a minimum. Regardless, I can hear the difference from across the room when his food goes into our dog’s metal bowl or on the floor.

It isn’t atypical to hear the clatter of a few pieces of his food hit the wood floor on the way and this night was no exception. As my daughter raced back to me with the now empty food scoop in hand, she says to me “I spilled some.” Now, I don’t know what some means to you, but it doesn’t mean the vast majority of the food, right?

Apparently “some” meant exactly that to my daughter. Certainly, as I looked down at my dog’s bowl there was an amount of food in the bowl and another amount outside the bowl on the floor. But “some” is not the word I would have used to describe the amount on the floor. “Most” is the word, if I have to be inexact about it, is the word that I would have used.

I know it’s a dumb example, but this is exactly why we need to measure things. All the words we have to describe portions are inexact. What does a few mean? More than 2 certainly, but is a “few’ deaths as the result of the millions of products you sold 3 people or 300 people or 3000 people? Compared to the whole, even 3000 out of a million might be a few!

The “majority” suffers this issue in news reporting all the time. When the majority of people approve of the President’s performance, it means some number greater than 50% approve. 50.0000001% is a majority. It’s not an overwhelming majority, but it’s a majority technically. And I’ve seen “most” used to mean a simple majority as well, which is crazy, since most clearly means something higher than that. Is 75% most? 80%? 90%? Who knows? The definition is variable.

What about “some”? Officially, it’s just a number greater than 0. Some of the food was outside the bowl. Indeed, not all of the food was outside the bowl, so some is a fair statement. But my version of some and my daughter’s version are really different in this case.

And it’s for this reason that English is an inexact language that true measurement is needed. A proportion would have told a much better story. Not that I would have expected my daughter to say “daddy, I spilled seventy-five percent of the food” but I can expect that from an adult.

Let’s talk real number in business. Put a scale to it – a proportion that is a problem, a count that is a problem, but some real measure of how much is really wrong. “We have some issues with code quality,” after seeing my daughter’s definition of “some” tonight, has a whole new meaning for me.

A friend of mine just recently came back from an Agile software development conference in Florida. Did you ever find it at least slightly odd that all conferences seem to be held in really nice places? Nobody ever says “come to a conference in Idaho.” Some part of me suspects that people don’t go to conferences to actually learn, but for the sightseeing/vacation opportunity.

Anyway, apparently Alistair Cockburn presented at the conference, and (I’m getting this second hand now) he showed a chart about how Agile improves knowledge gain…

It looked something like this:

Notice that it is unit-less on both axes. The cost axis you could almost forgive, since we have some intrinsic knowledge about what it means for something to cost more. Knowledge on the other hand… I don’t know how you measure that. I don’t think he means IQ in this case. His point was something along the lines of “with Agile, you gain knowledge about your work sooner.” His argument is essentially, with Waterfall you integrate and learn very late that things aren’t working.

The problem is, his chart is garbage. This isn’t scientific. He drew a picture. Anyone can draw a picture. There’s no study behind this. There’s no data about the impact to the customer about “learning late.” Even if his hypothesis was true and backed up by data, it has to have bearing on what your customer experiences to matter. Does late learning mean worse quality? Worse costs? Who knows… Not only is this chart imaginary, so is the impact. I drew a chart too (notice the flat line).

I was talking with one of my managers the other day when she presented an interesting point that I thought I’d share. There is a difference between knowing something and thinking you know something. Sure, there is pure ignorance, but it’s not the type of issue we were talking about.

The type of intellectual leap you shouldn’t make is the taking of incomplete knowledge and representing it as complete knowledge. It’s the overextension of a specific situation to being the general problem. For example, we know that some defects were introduced when the code was written years ago and are just appearing now, but that doesn’t mean that our problem is all latent bugs. This is inductive fallacy. All we’ve seen (for those that we can confirm at all) is that some bugs exist since the creation of the code, but this does not make the general statement “our issue is latent bugs” necessarily true.

The list of bad extensions is long, and yet easy to remedy. We should be talking about odds and probabilities and uncertainty and recognizing that it exists with much of what we deal with. For example, Stephen Kan, a researcher from IBM, recognized that test execution doesn’t happen linearly but instead follows and S-shaped curve. When we explored our patterns of test execution we discovered the same thing held true of us. And so, knowing that, we could set forth a realistic pattern of what would have to happen (for a given set of tests and duration to complete the testing) in order for us to be done one time. We tried it out, and found out that we don’t always perfectly follow the curve.

It wasn’t about perfectly following the curve. It was about more-or-less following it. There are essentially bands, plus or minus, around this idealized curve which represent the uncertain area which is normal variation from the perfect shape. As long as you stay within those, we feel comfortable that things are on track. But we represent the uncertainty, not ignore it, because it is there. The knowledge says “testing follows and s-shape”, but we don’t make that an absolute truth, just a general pattern that includes room for representing that we don’t understand everything that may cause it to deviate from that path.

"A little learning is a dangerous thing; drink deep, or taste not the Pierian spring: there shallow draughts intoxicate the brain, and drinking largely sobers us again."

- Alexander Pope, An Essay on Criticism

His quote is not just a warning that knowing a little causes you to do bad things, but that it is necessary for us drink deeply, acknowledge that we do not know everything and seek to figure out what it is we do not know.

Did I write this before? I can’t recall, but it came to mind today at work in a conversation I was having. From a user’s perspective, whether you intended the code that way or not, whether it is a “bug” (I didn’t intend it to do that and you don’t like it) or an “enhancement” (I did intend that, but you don’t like it), doesn’t matter.

The distinction we make on this topic is one of vanity. We want to somehow feel better for having done as we were told and yet not satisfied our customer. How is it ok to just be the order taker, to claim that we have done as asked, and that the change requested is an “enhancement”? In effect, “this is not my fault.”

It is your fault. It is OUR fault. We are one company, one organization, with a single customer (or group of customers, it doesn’t really matter) to serve. If we fail to serve that market, whether it is through coding incompetence (bugs) or failure to recognize the need (enhancements) doesn’t matter.

What matters is that we aren’t meeting their need, and beyond that, the distinction we make between bug and enhancement is only a small piece of information in the search for a preventable root cause. This isn’t a hard concept to grasp, yet we seem to struggle with it all the time.

If the customer doesn’t like it, it doesn’t matter the goodness of our intention. It must be remedied. The customer does not care how the undesirable behavior was introduced, just that it exists at all.

Because I tend to focus my comments on software development processes, I subscribe to a number of newsletters just to keep abreast of stuff that other people are writing. Generally, stuff is a rehash of what we’ve already heard – see my commentary on why it seems nobody is doing primary research anymore. Today I got an interesting one which was “100 IT Performance Metrics”.

I’ve decided to rename it to “100 metrics you mostly shouldn’t bother with.” Don’t get me wrong, I love metrics and data but there are just too many issues with this proposal from Mr. Spanos to ignore.

1. 100 is way too much information. It violates the idea of the critical few things you ought to control. And there are a critical few things which really make most of the difference.
2. Wrong statistic. Mr. Spanos suggests the average in a lot of places where the median would likely be more appropriate. See numbers 3, 4, 9, 10, 11, etc.
3. Rather than reporting one useful metric, he recommends many more than necessary. Boxplots, in many cases would better display the data than 3-4 graphs. For example, numbers 3 and 6 (mean and max resolution time) could be built into a single graph which also showed interquartile range, outliers and providing an easy to read month-over-month comparison. Just doing this could cut the number of proposed metrics in half or more.
4. Stratification hell – by type, by severity. Enough said. If you solve a problem and have fewer incidents the whole number will go down. It is highly unlikely that the resolution of high severity incidents will be replaced in equal volumes by medium severity incidents. Just measure the overall number, once.
5. Lack of scale. All these measures lack any sense of scale. Unlike a factory, the amount of work an IT shop is doing varies greatly from month to month. Without a normalizing factor, increases (or decreases) in any of these measures might entirely be due to changes in work volume.
6. Irrelevance. Average contractor cost? Here’s a place where median is a necessity, since one expensive contractor will blow this number out of the water. Also, who cares? Are you in the business of measuring what rate the market will bear or the rate of inflation year over year? If you’ve chosen to use contractors, you’re going to have to pay for them.
7. Unmeasurable. Change success rate. What the heck is a successful change? One that makes it to production? One that makes it to production with no bugs? One that yields no bugs once in production?
8. Lagging. All these metrics are after-the-fact. If you live by these metrics you have to wait for bad things to happen to you before you realize it. Find some leading indicators of when you’ll be off budget or off schedule or off quality targets and use those instead.

I could go on, but why bother. This is just too easy to pick apart. If you think implementing 100 metrics is the right thing to do, you’re off your rocker. 50 is probably too many. 25 is too. Think critical few things that make your business tick. I’d guess the number is probably 3-4 output metrics and 3-4 input metrics per each output – somewhere in the range of 12 to 16 measures should get you most of the way there.

I happened to be cleaning out my home office for some reason when I stumbled upon one of my old performance reviews. Given how dreadfully most companies conduct performance reviews, I usually pay little heed to the words written and just look at the dollars. It becomes a “putting your money where your mouth is” thing. Either the dollars match what you are saying or they don’t. If you say nice things but give a mediocre raise, then you can be sure that the words are hollow.

At any rate, I was (despite my tendency to just look at the dollars) re-reading the comments on this particular review. One of the comments was “… needs to tailor his presentation to the level of his audience. Senior individuals only want the conclusions… “

I have been well aware of the idea of tailoring to your audience for a long, long time. I think it’s one of those first thing you learn. Surely the most senior individual wants nothing to do with the dull details of your code, but what about the details of ones analysis? True, they probably don’t care about homoscedasticity in your data, but just the conclusions? That seems like the other end of the extreme – too little information. Is this really the right thing to do?

I think, if you ponder this proposal for a minute you will realize it’d be a foolish person who’d just come in with the conclusions. If I concluded to anyone, at any level, that “the world is flat” then I suspect the reasonable question they’d ask is “why do you think that is so?” This isn’t a question of tailoring to one’s level appropriately. No matter who you are, you can’t walk into a presentation with just the conclusions.

Why not? Any idiot (or liar) can write down a bunch of “conclusions.” In the absence of the data which shows why you believe what you believe, a slide of conclusions is worthless. Yes, there are nitty-gritty details of doing analysis, like how I transposed these columns of data into that pivot table in order to run my Mood’s Median test, which should be dropped from the presentation. But that’s the type of thing I wouldn’t explain to even the most detail oriented of my peers. Those details are irrelevant to everyone.

The issue isn’t what you should present to a senior individual, but whether what you have is worthy of any attention at all. Senior managers are still people whose brains work just like the brains of their subordinates. They need information to make decisions just as we all do, but they need information about different types of problems. If they don’t need to make a decision, what are you doing there at all presenting? And most importantly, if they asked for the information, then you can reasonably assume they wanted to hear more than the conclusion.

Bring along enough information to tell a complete story; far more than just the conclusions. If they just wanted the conclusion, you could send an email instead.

As I was sitting here doing nothing, or effectively nothing (I was playing mini-games on my computer), for some reason a bunch of failed change events came into my head. As my prior post would indicate, I fail to make change a lot of the time, though not for a lack of trying.

To be fair, if I make a change and it isn’t better, then I hardly count it as making a change. Sure, something is different but not in a way that is meaningful to our customer.

Anyway, I digress. I was thinking about change events and why they always seem to be such a big event. It brought to mind a story about trying to build a new requirements process. Years ago, I worked for a team who was doing a process redesign for their requirements process. It was going to be this big multi-day event where we brought everyone offsite to hash out a new way to do requirements.

Never mind the fact that I believe this type of work really needs to be done scientifically as opposed to just throwing people into a room and saying “design something.” Anyway, I was tasked to go off and get a resource from this one team to attend. I approached the vice president of the team and said something along the lines of “I’d like to have a representative from your team attend this 3 day offsite to redesign the requirements process.”

“No,” he replied. I sat there and stared at him. I really didn’t know what to say. It wasn’t an angry stare on my part, it was more like a mouth agape bewildered stare. Kind of like the way my dog cocks his head to the side when something makes a funny noise. I’m pretty sure that’s what I looked like. Fortunately, he went on.

“We already redesigned this process once with Bob [a
coworker of mine]. Why should I send resources to do it again?”

Fair enough. It was true; Bob had tried to do this process redesign before. It had failed not because they were unable to come to an agreement over the change but because the team who was supposed to adopt the change refused. Foolishly, Bob had neglected a major stakeholder in the process. But, darn it all, we weren’t going to make that mistake again.

And yet we did. The mistake wasn’t trying to get all the people in the room, or doing it via an offsite, or doing it without understanding the existing process fully. The mistake was making change an event. An event is easy to resist. And event has a clear beginning and thus a clear line in the sand you can take a stand against.

What if change just snuck up on you? It happens all the time in our regular lives and we handle them in stride pretty well. Tools like twitter, Facebook, even the internet and email just creep into our lives without making a big scene. It’s hard to resist something that shows up quietly, that doesn’t really come knocking or announcing itself. It just shows up because it is better and more useful. Now that we have these things, we can’t imagine life without them.

Why can’t we approach process change this way? What if instead of a big-bang event to redesign the requirements process we just made some small quiet change to the way we did things? What if we just started inviting teams to JAR sessions. No training, no announcements that this was the new way to do things. Just invite them on some projects and see how they feel about it. If they like it, maybe we do it on some more projects. Eventually we’re doing it all the time and it becomes an accepted way of doing things.

There’s no real need, except potentially for speed purposes, to announce a process change. And given the enormous resistance that can be placed against an event, perhaps it is just better to sneak a little change for the better here or there. Do it often and eventually the process will be changed for the better, and significantly so.

Perhaps a better title for this posting would be “eventless change,” but the idea is the same. Change doesn’t have to be eventful for it to be impactful.