Information work rewards hoarding

While I don’t think it has to be the case, being an information worker rewards hoarding behaviors. Or, more specifically, and probably why Deming was opposed to performance reviews, the only way you can differentiate yourself in information work is to not share what you know. And that’s the exact opposite of what the company wants.

We’ve had several failed attempts to create knowledge communities around the office, and I suspect some of the reason is nobody wants to share any information. Why? Well, if I told you what I know, what would differentiate you from me? In a manufacturing environment or sports, some of what makes me different is something that you can’t replicate – it’s my intrinsic ability to do a task better than you. I may have greater hand-eye coordination or greater speed or constitution, or all of the above. But with knowledge, it’s a bit like having a piece of digital music. It’s easily copied and passed from one person to another.

If you know what I know, the company may be better off, but by bringing you up to my level (giving you my ideas), I stand to eliminate the differentiation between myself and you. So, why would I want to give you my information? That’s not to say we should behave like group-think; that type of design-by-committee is destructive to the search space to solve a problem. If we all work together then we all work together in the same part of the world of potential solutions. Someone has to be thinking about something totally different in order for new ideas to emerge.

The question remains, how do we get from an environment where I’m not incented to share what I know to an environment where I’m rewarded for having (and sharing) new ideas.

The above average driver

According to some studies, nearly 65% of vehicle drivers rate themselves as “above-average” drivers. These studies cite “optimism bias” as the cause and (having not read the studies) it may well be the case. People simply think they’ll do better than the other guy.

On the other hand, another simpler explanation might exist. What does average mean to you? For someone like me, who happens to live statistics every day, average (or median) means something very specific. However, I think if you went out on the street and asked a question like “are you an above average, average or below average driver?” that most people would translate that question into their heads as “are you a good, so-so, or bad driver?” The issue simply becomes one of what people interpret average to mean. I wouldn’t cop to being a bad driver. Plus I lack perspective to know what an average driver experiences? Are we talking accidents per year? Speeding tickets per year? I’ve never had a single moving violation in my entire driving career to date, but I’ve been in 1 accident where the car I was driving was totaled. So where exactly does my skill land? I don’t know. I’ve never seen data about what other drivers experience.

Let’s say the “average” driver has 1 accident in their lifetime. I know that’s not true, but I’m just proposing it to illustrate a point. If there was extremely low variation in the population (and it were normally distributed), above average drivers might have .9 accidents in their lifetime while below average drivers have 1.1 accidents in their lifetime. It might be a statistical difference, but it’s hardly a practical difference. Being below average in such a population doesn’t mean much at all.

At issue here isn’t what the “average” statistic really means, but what it means to people when they hear they are above average or below average. Being below average to people means something really bad, when, in fact, a below average performance might still be more than acceptable. We equate anything below-average as unacceptable, which it doesn’t necessarily have to be.

For example, take a professional baseball player, put them on a team of little league players, and even their lowest “below-average” day would far surpass the skill of every kid on every team in the little leagues.

Ok, so what’s all this got to do with anything I write about? Don’t take for granted that people know what you mean when you use words like “average”. Average is a measure of central tendency, but there’s a more colloquial definition which means mediocre. Thus, having a below average performance might not mean to people “a statistically different performance compared to the larger population” and instead means “we suck.”

Setting a target

I have blogged before about something like this topic, but it came up again recently and I had some additional thoughts. The last time I discussed setting goals against industry baselines my point was that it isn’t necessarily using the industry data if the performance of the industry at large is unsatisfactory. That’s especially true in software, where study after study has concluded that our ability to delivery what is requested on time and on budget is particularly poor.

At any rate, the topic came up again at work about whether we should set a baseline goal against industry data on performance. When you consider what statistics are commonly available, a problem begins to emerge. The mean and median both represent the central tendency of the population. Now, don’t get me wrong, used properly, knowing this data is critical to making informed process change.

But my issue isn’t with the calculation so much as the mean (or median) as being the goal. If you are comparing yourself against the industry baseline, where should you set your target? Should you set your goal based on the population median? If you meet that goal, what would we conclude? That you are a very average company is what I’d conclude. And who wants to do business with an average company, however you might measure that?

If you are going to get an industry baseline, then it should only be to understand what kind of performance you need to become a world class organization. The number I’ve heard bandied about is “better than 95%” of all other companies. Is that necessary? I’m not really sure. 2 standard deviations out seems good. Certainly it’s better than being right in the middle of the pack.

It really doesn’t matter how you are performing against the industry today. If you are among the worst, in the middle, or even among the best, you don’t want to set a goal (or sit on your laurels) of achieving average results. The saying isn’t “build an average mousetrap and the world will beat a path to your door.” Nobody goes out of their way for average, so don’t set your goals against the industry average. You need to do far better than that.

How is this different from what you do today?

Recently I’ve been attending, as sort of a consultant, a process redesign effort in another part of the company. Coming in during the middle of the process, it was hard to see what had already been done while trying to keep up with a still-moving project. As I listened in, part of my confusion came about because I couldn’t tell if the group was talking about the new process they were designing or the old process.

It wasn’t an issue of not knowing the lingo or seeing the process flow but because the team would weave in and out between the two processes, like they weren’t even different at all. So finally, I piped in, “how is the new process different from the existing process?”

Someone answered “well, we don’t have an existing process.” And someone else responded, “yes we do, it just varies project to project.”

But listening to their new process, it was clear they were prepared to allow significant variation in the new process as well. There was talk about allowing the team to choose the artifacts created, to choose which steps to skip over, and so on. I pressed them on what was really different.

“Do you have templates for the new documents you want created?” Reply: “well, there aren’t any new documents, we already do these documents today.”

“Do you have role definitions for all the new job roles?” Reply: “well, we already have all the roles today.”

“Do you have a process flow which shows the delta between the new and old?” Reply: “we have a new process flow.” Me: “but it’s so high level couldn’t you fit your old process into it?” Reply: “yes, well…”

Ah ha! What exactly are we talking about here? If we can’t see the difference then what exactly is different? As I continued to probe I did find out that they wanted to be more “collaborative” in the process. Now that’s not a bad thing, but it sure is hard to put that down in a process. Indeed, there’s no reason to believe they weren’t collaborative in the old process. I started to wonder if they were solving anything at all. The whole thing was starting to seem very pie-in-the-sky. So little was actionable or different than what they were doing today.

In the end I said to the team “you need a before and after view of this process change.” Sit down, write down what you are (more or less) doing today and then annotate all the changes you are making. In their case it seemed what they were proposing was simply more meetings. If you find that the new process fits nicely into the old process flow you need to be thinking long and hard about what you are really doing differently.

Process redesign fail.

No more RACIs

I’ve decided that I hate RACIs. There’s just too much wrong with them to make them worth doing. A RACI, in case you are unfamiliar (or RASCI sometimes) is a table which lists tasks along one axis, people (or job roles) along another axis and at the intersection marks someone one or more of R,A,S,C,I (responsible, accountable, supporting, consulted, informed).

Now, depending on the school of thought you subscribe to, the S option may or may not exist. A simple RACI might look something like:

	Mommy	Daddy	Daughter
Wake up	R	R, A	R
Get dressed	R, A	R	R
Go to school	A	I	R
Ask about school day	I	R	C
Make dinner (at least in my house)	C	R, A	I
Have dessert	C, A	I	R

One can thus determine what my day is like. In the morning, we are all responsible for waking up, while if someone doesn’t get up on time it’s Daddy’s head who is going to roll. Then, we’re all responsible for getting dressed, but since Daddy has no fashion sense, it’s Mommy who is accountable for this task. I’m simply informed about our daughter going to school since I’m going to work. Technically, though my daughter is too young to drive herself, she’s the one responsible for going to school while Mommy is the one who gets in trouble if my 3 year old doesn’t show up. I’m responsible for asking about her school day, Mommy is simply informed. She overhears my daughter say (who is only 3 but clearly has the kid role figure out) in response to “what’d you do at school today?” almost always replies “nothing.” Obviously, my daughter being the expert on her school day is consulted on this task. One might also choose Supportive I suppose since I can’t complete the task without her. Dinner is strictly a Daddy job with Mommy having some say about what I make for dinner and the little one just eating what is presented. If we consulted her every night would be chicken nuggets, pizza or noodles. And finally for dessert, the little one is responsible for the eating while Daddy is simply informed. Mommy has the go/no-go decision on this one.

Phew. Now that is one heck of a lot of data all from that little table, and this is a joke of a RACI. No real job that I do is so basic as to have just 3 players and 6 tasks to it. You can easily see that many more tasks might be appropriate even in my daily plan – including showers, dishes, bedtime for my daughter, etc. We actually follow a pretty predictable pattern each day – a process you might say – and yet there’s enormous detail that could go into it. The fact of the matter is that any realistic RACI would get unmanageable and quick.

The second issue I have with RACIs is that they’re a punt for figuring out a good process. If you have to go to all this trouble to figure out who does what in the process, isn’t it a reasonable question to ask if you might have a really un-lean process that simply needs fewer actors or individuals who actually have the authority to both be responsible and accountable for the task to get done? The mere fact that R (the responsible person who does the work) can be separated from the A (the accountable person who gets in trouble if the work doesn’t get done) is crazy! The idea that we’d consult all kinds of people suggests that we don’t have enough knowledge (or the wherewithal to gather it) about the job we’re tasked with doing. And informing? Look, if the person isn’t going to act on the data you are provided (ie, give feedback, which makes the consulted not informed) then why are you giving them the information!?!?

Third, nobody gets these things right anyway. Can more than one person be accountable? Responsible? I think not. And yet, they’re always a mishmash of several people being responsible for the same task. Someone has to be running the show and the moment you allow everyone to be accountable then nobody is accountable and the same goes for responsibility. It just becomes a giant circle of finger-pointing when the task doesn’t get done.

Finally, these things are usually more of a suggestion than a reality. Putting that much effort into something that isn’t going to be followed is insane. Even in my own house, with the six silly tasks that I have, if I’m working late then Mommy becomes responsible for all the tasks that I would normally be. Life marches on, regardless of what my RACI says ought to happen.

Don’t do RACIs. Do a nice swimlane process flow and leave it at that.

Far fewer soft benefits

Recently I was presented with an ROI for some project.  It included some costs, which weren’t that interesting and it included one hard benefit which was a time savings.  And then it included a series of soft benefits including improved quality, improved customer satisfaction, etc.

Now, as far as I understand it, a hard benefit is one which you can associate a dollar value with and a soft benefit is a benefit which has no dollar value.  But looking above, these benefits aren’t soft benefits.  They’re (potentially) hard benefits.  Quality is worth something since it costs money to fix defects.  Customer satisfaction is worth something because unhappy customers don’t spend their money with you.

These aren’t soft benefits at all.  They’re hard benefits that you didn’t bother to figure out!  But don’t confuse them.  Just because you haven’t figured out the value is very different from having no value.  I’ve run into just one benefit of a process change which was truly a soft benefit.

It was the benefit of feeling more comfortable about what we were delivering.  For example, doing regression testing and finding no bugs has no hard benefit.  Peace of mind that the code is correct is a soft benefit.  Your customer doesn’t care how well you sleep at night and you didn’t prevent any bugs so you had no impact on quality or customer satisfaction.  That peace of mind is a soft benefit.  Everything else is a hard benefit that you don’t know the value of.  If you’re going to mention it in an ROI, figure out what it is worth.  If the change is so small you can’t measure it, then don’t list it as a benefit.  But lets not pretend here you are getting a “soft” benefit when you are truly getting is a minuscule hard benefit.

The consumer is always right

I was sitting through a meeting the other day and we were talking about the new test case management system that we were installing. It’s nothing special, but one of the features it offers is the ability to track defects along with the test cases. The thing is, as a company we already have 3 other defect tracking systems from different vendors, none of which are integrated with our test case management system. True, it is an annoyance and certainly adds some cost to quality assurance to record the defect when you have to go into a different system.

However, the issue I have is that quality assurance designed the workflow and fields that would go into the defect tracking system. Quality assurance is a (sort-of) user of the tool. They have to enter data into it, so making it convenient for them to enter data is a good thing. However, they aren’t the primary consumer of the data – development is. The reason we enter bugs into the defect tracking system is (besides metrics) so that development gets a detailed account of what went wrong so they can fix it. Although we supply defects into the process, it is development that has to consume our input and use it.

So why then would you have QA define the input for the team who needs the data? It is illogical that QA should determine what information and in what format they’ll be giving it to development. Development, who needs adequate input to fix the bug, should be defining what inputs they need. QA could input more data than development needs if they felt they needed some data recorded too, provided it isn’t interfering with what development needs to get their job done.

Now, maybe development requests something that is ridiculous and QA ought to negotiate for a more acceptable request, but generally I believe that it is the consumer who gets to define what they need from their supplier. In our case, we didn’t ask development at all what they needed, we just built something. We didn’t ask development if they’d be willing to use our new defect tracking system.

I think, if it hasn’t been set out as a universal truth somewhere else before, that it ought to be done so now – the consumer defines the required input, not the supplier.

Why measurement is necessary

This evening, my daughter, in an attempt to be helpful offered to put our dog’s food into his bowl for dinner. She does this every now and again, and usually the mess is kept to a minimum. Regardless, I can hear the difference from across the room when his food goes into our dog’s metal bowl or on the floor.

It isn’t atypical to hear the clatter of a few pieces of his food hit the wood floor on the way and this night was no exception. As my daughter raced back to me with the now empty food scoop in hand, she says to me “I spilled some.” Now, I don’t know what some means to you, but it doesn’t mean the vast majority of the food, right?

Apparently “some” meant exactly that to my daughter. Certainly, as I looked down at my dog’s bowl there was an amount of food in the bowl and another amount outside the bowl on the floor. But “some” is not the word I would have used to describe the amount on the floor. “Most” is the word, if I have to be inexact about it, is the word that I would have used.

I know it’s a dumb example, but this is exactly why we need to measure things. All the words we have to describe portions are inexact. What does a few mean? More than 2 certainly, but is a “few’ deaths as the result of the millions of products you sold 3 people or 300 people or 3000 people? Compared to the whole, even 3000 out of a million might be a few!

The “majority” suffers this issue in news reporting all the time. When the majority of people approve of the President’s performance, it means some number greater than 50% approve. 50.0000001% is a majority. It’s not an overwhelming majority, but it’s a majority technically. And I’ve seen “most” used to mean a simple majority as well, which is crazy, since most clearly means something higher than that. Is 75% most? 80%? 90%? Who knows? The definition is variable.

What about “some”? Officially, it’s just a number greater than 0. Some of the food was outside the bowl. Indeed, not all of the food was outside the bowl, so some is a fair statement. But my version of some and my daughter’s version are really different in this case.

And it’s for this reason that English is an inexact language that true measurement is needed. A proportion would have told a much better story. Not that I would have expected my daughter to say “daddy, I spilled seventy-five percent of the food” but I can expect that from an adult.

Let’s talk real number in business. Put a scale to it – a proportion that is a problem, a count that is a problem, but some real measure of how much is really wrong. “We have some issues with code quality,” after seeing my daughter’s definition of “some” tonight, has a whole new meaning for me.

No good data? Make it up

A friend of mine just recently came back from an Agile software development conference in Florida. Did you ever find it at least slightly odd that all conferences seem to be held in really nice places? Nobody ever says “come to a conference in Idaho.” Some part of me suspects that people don’t go to conferences to actually learn, but for the sightseeing/vacation opportunity.

Anyway, apparently Alistair Cockburn presented at the conference, and (I’m getting this second hand now) he showed a chart about how Agile improves knowledge gain…

It looked something like this:

Notice that it is unit-less on both axes. The cost axis you could almost forgive, since we have some intrinsic knowledge about what it means for something to cost more. Knowledge on the other hand… I don’t know how you measure that. I don’t think he means IQ in this case. His point was something along the lines of “with Agile, you gain knowledge about your work sooner.” His argument is essentially, with Waterfall you integrate and learn very late that things aren’t working.

The problem is, his chart is garbage. This isn’t scientific. He drew a picture. Anyone can draw a picture. There’s no study behind this. There’s no data about the impact to the customer about “learning late.” Even if his hypothesis was true and backed up by data, it has to have bearing on what your customer experiences to matter. Does late learning mean worse quality? Worse costs? Who knows… Not only is this chart imaginary, so is the impact. I drew a chart too (notice the flat line).

The difference between knowing and thinking you know

I was talking with one of my managers the other day when she presented an interesting point that I thought I’d share. There is a difference between knowing something and thinking you know something. Sure, there is pure ignorance, but it’s not the type of issue we were talking about.

The type of intellectual leap you shouldn’t make is the taking of incomplete knowledge and representing it as complete knowledge. It’s the overextension of a specific situation to being the general problem. For example, we know that some defects were introduced when the code was written years ago and are just appearing now, but that doesn’t mean that our problem is all latent bugs. This is inductive fallacy. All we’ve seen (for those that we can confirm at all) is that some bugs exist since the creation of the code, but this does not make the general statement “our issue is latent bugs” necessarily true.

The list of bad extensions is long, and yet easy to remedy. We should be talking about odds and probabilities and uncertainty and recognizing that it exists with much of what we deal with. For example, Stephen Kan, a researcher from IBM, recognized that test execution doesn’t happen linearly but instead follows and S-shaped curve. When we explored our patterns of test execution we discovered the same thing held true of us. And so, knowing that, we could set forth a realistic pattern of what would have to happen (for a given set of tests and duration to complete the testing) in order for us to be done one time. We tried it out, and found out that we don’t always perfectly follow the curve.

It wasn’t about perfectly following the curve. It was about more-or-less following it. There are essentially bands, plus or minus, around this idealized curve which represent the uncertain area which is normal variation from the perfect shape. As long as you stay within those, we feel comfortable that things are on track. But we represent the uncertainty, not ignore it, because it is there. The knowledge says “testing follows and s-shape”, but we don’t make that an absolute truth, just a general pattern that includes room for representing that we don’t understand everything that may cause it to deviate from that path.

"A little learning is a dangerous thing; drink deep, or taste not the Pierian spring: there shallow draughts intoxicate the brain, and drinking largely sobers us again."

- Alexander Pope, An Essay on Criticism

His quote is not just a warning that knowing a little causes you to do bad things, but that it is necessary for us drink deeply, acknowledge that we do not know everything and seek to figure out what it is we do not know.