The Hawthorne Effect of constant change

I was having a great email chat with my former vice president the other day.  The original conversation was about how a process improvement team should be organized and we got onto the topic of the virtues of a centralized or decentralized model of process ownership.

While centralization seems to make sense, I argued that the change management challenges created by disassociating the improvement work with the teams who had to improve wasn’t worth the added value of defining common processes across silos.  In fact, in our organization, the silos don’t share that many common processes, so even if centrally defined, little would be shared.

From there, as our conversations often do, we wandered over to Agile software development.  My friend is a huge proponent, while I am an outright non-believer (as if you couldn’t tell from my other posts).  And the discussion became about whether one of the main tenets of Agile – that the team decides the process – was a good idea.  In a separate post, I’m going to tell you why no team should ever be allowed to decide the process for themselves, but let’s ignore that for the purposes of this post.

For whatever reasons, valid or invalid, having a team decide their own process makes me uncomfortable.  It may be that I’m a control freak, but something is unsettling about it.  However, over my years I have mellowed from a strict dictator about process compliance to someone who is willing to take some input (the operative word here is some).  So I conceded that I could be supportive of a Toyota-like model, where those who want to improve the process have to prove its value and cause the change to be socialized with the entire organization.  Local sub-optimization is not acceptable.

Well, I don’t know about Toyota and their proof process, but it’s certainly better than the appearance that Agile gives, which is the team collectively decides what the right process is.  No data, just gut decisions.  And then it struck me – the act of allowing change may simply be all you need to create superior performance.  It has nothing to do with the quality of the change!  It’s the Hawthorne Effect at work.  By allowing teams to control their environment, they get the value, the excitement, of trying something new.  And as a result, if they like it, it becomes part of their process.  The value of that particular change may fade.  In fact, it may even be damaging, but offset by the value of enthusiasm.  At least, until the enthusiasm for that change ends.  And then they make another change, which has the same short term benefit, and another, and another and another.  Each change is good or bad, or hopefully evenly distributed, so that once the Hawthorne Effect fades any damage the change might really do is offset by some positive real change that was also made.  Not that you’re really getting anywhere ever, one step forward, one step back.

Of course, I wonder if the constant change eventually loses its magic, the act of improving becoming no longer exciting.  And then of course, all those steps backward will come to haunt you.

Software defects shouldn’t be measured per unit

So I had this (crazy) thought the other day.  Why are we measuring software in defects per unit delivered?  Doing so treats software like nuts and bolts, a relatively homogeneous thing.  Software is more like a lawnmower or a car, and while delivering such a thing in working order is important, we don’t view a car as satisfactory if we drive it home and it bursts into flames in the driveway.

Reliable software is like a reliable car – frequently used software has many more opportunities to fail than software that is never run.  In fact, I think the key measurement of how good the software is, isn’t how many defects the software has, but how many defects it exhibits per line of code executed or feature run.

You see this thought process in the way developers complain about the bugs QA reports – “sure, it breaks if you stand on one leg, when the moon is full, and you simultaneously press 36 keys on the keyboard.”  It’s a valid point, a defect that will rarely manifest itself (assuming it doesn’t result in the death of the user), isn’t as important as a defect that will cause the system to crash every time it boots.

And yet, while we give defects a severity, it’s really not the same as conducting an FMEA on the error.  We only talk about the risk (severity) if it occurs, and ignore the probability and detectability of it occurring.  And yet, in real life we make all kinds of decisions taking more than just severity into consideration.  I still get into my car and drive to work despite the risk that a tire might blow out thus killing me on the highway.  And despite the fact that I wouldn’t be able to detect it was about to happen until it actually happened.  Probability matters.

So here’s my simple objective proposal.  Instead of measuring defects and trying to rate them, since we are miserably bad at that, let’s measure defects in terms of defects per unit executed.  I’ll leave it up to the reader to argue about whether line of code or function point should be used as the unit measure.  It doesn’t really matter.

If you’re going to ask for feedback…

So I did something today that I shouldn’t have done.  See, a peer of mine asked me to present on the project I’m working on.  I’ve given similar presentations a few times now, mostly to senior executives, so it’s a bit old hat for me.  Bored with the idea of rehashing yet the same presentation to my peers, I decided I’d come up with a new presentation.  One that covered a different topic than the usual. 

Typically the presentation is about what we’re working on and how it’s going.  Instead, I decided I’d give a presentation on the team organization and how it was working.  This isn’t your typical team at all, which would be a really boring presentation.  This team is a process design team, and they’re one of the first in the company.

But I digress, the point is, I made up a very brief, four slide presentation.  The first slide was ‘how does the process work?’, the second slide ‘what works well about it,’ the third slide ‘what doesn’t work,’ and the final slide is ‘what have I done about the things that don’t work.’

Anyway, I cobbled this together in an hour or so and then sent it off to the requester for inclusion in the larger slide deck.  Foolishly, I put in the email message “feedback welcome” or something like that.  I really was just being polite.  But, of course, the recipient of my email took it as a real invitation and sent me feedback.  Minor stuff, really, but since I had no expectations of having to make edits my initial reaction was to ignore him.  In fact, I was kind of irritated that he nit picked at my work!  I mean really, what is the point of telling me that “starbursts look better than rectangles” for the little overlays I did?  Honestly!

And fortunately, that’s when my better judgement kicked in.  Now, in this case, if I had just ignored him, there probably would have been few ill effects.  But, generally speaking, it’s a change management disaster to ask someone what they think and then ignore them.  People want to be heard, and in fact, a powerful change management tool is to do just that.  And the best way for them to know that they’ve been heard is to incorporate their feedback into your work.

Guess what, most people really don’t have a lot more than nitpicks, at least until you start ignoring them.  Then they get mad and nitpicks turn into finding major issues with your work.  You really, really don’t want to get to that stage.

How do I know?  I’ve done it to people.  They’ll be giving a presentation for review and I’ll suggest some minor wording change on the first couple slides because I think it’ll read better and the presenter will say something like “I like it as is.”  Suddenly I’m finding myself very annoyed.  I’m not being listened to!  So when they finally get to the meat of their presentation, then I rail on them about the quality of their data, their lack of hypothesis testing, lack of measurement system analysis, and so on.  I will rip the rest of the presentation apart.  And I won’t stop there.  I will get my hands on their data, and I will run my own tests on it and I will publicly present my findings.  Oh yes, I’ve done it.  And then how will you feel having to explain my conflicting (but correct) assertions about your data?  And you could’ve just changed the few words on the first slide, now couldn’t you?

Hey, at least I’m self-aware enough to recognize my behavior and admit to it.  Most people aren’t and that means you ignore them at your own peril.  Frankly, even ignoring people who are aware of their frustration is a bad idea.  Asking for feedback is good, except if you don’t act on it.  Then it’d be better if you hadn’t asked at all.  Nobody likes to be ignored.

Little breakthroughs

So for all the time I’ve been writing about process and related topics, I never thought I’d be thankful for the little revelation that I’ve now heard twice in as many days.  I speak of the most basic of premises, which, to quote one of the two people I heard it from – “we should be doing things that solve the problem.”

And they said it with a straight face.  What have you been doing all along!?!?!?  Deliberately NOT solving the problem?  Ah, how productive we feel when we’re getting stuff done.  Who cares if it is the right stuff, that’s irrelevant.

I don’t think it’s malice so much as a bad game of operator.  It goes like this.  Something goes wrong in production and so a client picks up the phone and calls the president of the company and yells at him.  The president calls the CIO and yells at him (stuff rolls downhill, you know).  The CIO calls the head of QA and reams him a new one.  And the head of QA, now being at a low enough level to actually make some action take place declares that X will now happen, because he thinks that X will solve the problem.  It’s irrelevant whether X will actually solve the problem.  But ignoring why he thought X would solve the issue, he directs one of his managers to make it happen.  We’re now 4 levels removed from the person with the problem, and yet the head of QA is willing to make something happen.  Something!  Anything!  For god’s sake, the client is unhappy!  Do something!

And something we do.  Does it solve the problem?  Who knows?  The guy doing the something doesn’t know what the problem is anymore, just what the solution is.  Indeed, it is a little miracle when the guy doing that something finally asks “why am I doing this?”  We are not an army, we do not hear “jump” and ask “how high”.  You’re not paying me, or any of us, to simply do your bidding.  If you are, you are overpaying us all a great deal.

It’d serve us all well to remember, that the things we do should directly relate to solving a problem.  If the thing appears to solve no problem, then we should be asking what the problem is that we’re solving and intelligently recommend a faster, better way to solve the issue if one exists.  After all, we’re here to solve problems, not just blindly do the bidding of others.

I took something I thought was obvious for granted.  Little did I know that the first thing the guy furthest down the chain would need to know is the most basic of assumptions - what we are doing should solve a problem.

Looking good or doing good

As Yoda said “do or do not, there is no try.”  I was listening in on a meeting when I realized that it was all about looking like we were doing, but we weren’t actually doing anything.

In this particular meeting, it kicked off by restating the goals that the organization had.  That’s a good thing.  But the very next slide listed 51 initiatives we had to try and meet those goals.  And the speaker went on to show how much stuff we were doing, sliced by the department being affected, the issue type being affected (cost, quality or speed), the types of defect root causes expected to be affected, and more!   The same 51 initiatives, which are probably too many to begin with, shown in all kinds of pie charts about how much was being done.

What was the point of this meeting?  Looking good.  It was a “hey, we’ve got lots of stuff going on!” but not necessarily “hey, we’ve actually been capable of making a difference.”  I can see why you’d hold a meeting like this.  If you can’t actually get anything done, at least you can justify your existence by indicating that you are trying to get things done.

Here’s the issue.  ‘Try’ will not save the company, only doing it, actually making the change for the better, will.  51 active initiatives is trying at a lot of things and really doing absolutely nothing.  Spinning your wheels, churning the data, presenting charts does not change the organization.  When people hold CYA (cover your a**) meetings, send CYA emails or make CYA presentations, it’s because they need the cover.  Forget the CYA.  Stop trying to look good, it’s a natural byproduct of actually doing good.  Unfortunately, looking good is easier to do, but the smoke will clear.

The only decision you can control

Seth Godin’s blog is all about marketing, but something he wrote a couple days ago reminded me about an important aspect of change management.  He rightfully responded to a critique of his work The Dip wherein the critic assessed that luck may have more to do with success than the decision to press on.  I’ll freely admit I haven’t read Seth’s book – I’m not much of a reader (in spite of all the writing I do).  I hope I’ve gotten the gist of his work based on a friend’s review.

But the key takeaway is this: you don’t have to be a pawn in the game.  When it comes to getting someone who doesn’t want to do something for you, the only person you can control in the decision making process is, in fact, yourself.  I know that approaching the problem of someone who doesn’t want to do what you want often feels like pushing a piece of string uphill.

But there’s a great, and sarcastic, quote that pretty much summarizes the problem – “Have you considered that the common element in all your failed relationships is you?”  (Sadly, I don’t know who to attribute this quote to, someone please let me know.)  Not unlike luck, there are many factors in every relationship that you have no control over.  You can’t control the other party in any way shape or form – not their mood, not their motivations, nothing.  I don’t care if you’re their boss or parent or captor.  Even if you can literally twist their arm behind their back and force them upon the threat of pain or death to do your bidding, you can bet that the compliance you will get at best will be malicious compliance.

Nobody has ever credited the mindless armies of zombies in movies with being thoughtful pursuers of the hero or heroine.  They can’t even figure out how to open doors most of the time, or at least they’re not motivated to figure it out.  Zombies don’t make great employees.

Even in all the things you can’t control about a relationship, there is one thing that you have complete control over – yourself.  You can choose to argue with the person you’re trying to convince.  You can get defensive.  You can stomp your feet and demand.  You can even rid yourself of the person, but no matter how you look at it, you’ve failed to get what you really wanted.  If you didn’t convince the person, eliminating them shouldn’t be counted among your successes.

Instead, it falls upon you to be the bigger person by being the smaller person.  Put your interests and goals second and find a win-win for you and the other party.  Appeal to them in a way that works for them – their ego, their power, their job satisfaction.  It has to apper to be their choice. 

Cynically, yes, you are manipulating a person to get what you want, but if both of you come out happy in the end, I’m not sure you’ve done too much harm.

A solution looking for a problem

Knowledge Management rubs me the wrong way.  The idea seems simple enough, so simple that it appears obvious and good – if people share their knowledge they will make everyone more effective at their jobs.

It probably offends me because it’s almost the exact opposite of process orientation.  I’m sure in reality that they’re quite compatible, but the place it’s coming from feels wrong to me.  I happen to believe that people are inherently lazy.  There’s even some evidence now that it might be genetically encoded in us to be that way.  Of course, we also see lots of evidence for this in behavioral economics.  The Pension Protection Act took advantage of people’s inherent laziness to help them save for retirement.  Rather than having people opt into their 401k, they can now be automatically enrolled and have to opt out instead.  Surprise, surprise, enrollment rates went through the roof AND people stayed in the plan.  We are built to be lazy.

So, with that in mind, the idea that people who have formed these ‘informal networks’ will be willing to volunteer their time to write down everything they know and submit it as a ‘knowledge asset’ just doesn’t line up with my belief system.

Worse than that, the value of Knowledge Management (KM, hereafter) is difficult if not impossible to measure.  In theory, every employee saves time be reusing knowledge that already exists – if only they can find it effectively.  But since the effect of KM isn’t detectable, KM people talk about about the key X’s that make KM successful – engage, reuse and something else I can’t remember. 

It doesn’t matter anyway.  It’s the first time I’ve seen a project proposal come with X’s and no output measurement (Y).  It’s very typical for someone to give you the end result – Y millions of dollars saved a year or Y% reduction in cycle time.  Often they can’t tell you what about their process actually makes the difference – whether it’s the co-location (I only bring that up because I just read an interesting article which showed that co-location by itself might account for a majority of the benefits that Agile folks claim) or the peer reviews or who knows what.

But no!  Knowledge Management is a solution looking for a problem.  We’re told what makes KM work well, and yet we can’t tell what effect it has on anything.  Honestly, even if I don’t know why it works, thus causing me to have to engage in a long study to figure out what a given process has value, at least you can compare it to other processes and figure out it HAS value.  Not so for KM – they mostly skip over that part and say ‘trust me, it’s good for you.’  I think I’ll pass.

Standards for the sake of standards

I’ve been forced to rewrite a large section of this post, since my original optimism about the value of controlling process variation has been dashed to the rocks by a healthy dose of reality.  I have no idea what I was thinking when I first wrote this.

Just the other day I was in a meeting with a bunch of business folks and some people from our team.  Some time back we set out to standardize the requirements gathering process.  Over the course of a few weeks we researched the industry standards and built a model for requirements gathering centered around some very basic ideas like JAR sessions and clear, concise requirements.  Certainly there was nothing that was brain surgery about it.

As we neared the end of our pilots, which people seemed very happy with, we realized we needed to plan for ongoing support for the process.  One of the key features, right or wrong, was that every project would have a trained facilitator for the JAR sessions.  For the sheer volume of projects we do every year, we’re talking about a significant resource expense to have these people on every project.

When we went back to the sponsors and said “hey, we need X millions of dollars to pay for resources over the next 5 years” to support our standard process, it didn’t go over that well.  Their response was, logically, “and what’s the benefit of paying for these people?”

Ah, how much I love it when the rules of the game change.  We’d now gone from standardize to improve.  At least in as much as the improvement had to pay for the resources we were asking for.  What to do, what to do?

I think panic set in at this point.  Standardized was OK with our sponsor as long as it was free.  Since we were asking for resources to support the standard, it suddenly wasn’t appealing to just have a standard.  And now we were being asked to prove that it was worth making the investment.  We’d never planned on it being worth anything, but just that it’d become the basis for future improvements.

The problem is, controlling variation doesn’t add a lot of value when the cost of one project can be offset by the value of another.  In the old process, we used to spend an average of 11.28% of our project money on requirements.  It had a standard deviation of about 7.3%.  In the new process, we’d spend an average of 11.3% of our project money on requirements and it has a 1.7% standard deviation.

Of course, having told you all this, the standard deviation is not meaningful in this case.  Recall from a prior post that the average describes what you’ll encounter over the long run.  When it comes to money, something funny happens.  If you overspend on one project and then underspend on another, it can all come out in the wash.  It’s different from say, building widgets, where if you build it too large or too small that it’s defective.  Money is a different animal and it’s likely that your CTQs will be around average cost, not variation in the process.

Think about it, let’s say you’re offering some service to your customers.  On average it’s costing you $100 to deliver the service.  If it sometimes costs you $10 and other times costs you $190, that’s fine.  As long as it averages out to a profitable number, you’re good.  You can charge every customer something over $100 and even though you’ll lose some money some of the time, you’ll still be turning a profit in the long run.  And if you can’t make money at an average cost of $100, then controlling the variation of the process isn’t going to make the situation better.  You HAVE to move the average.

Anyway, back to the problem.  Since the new process had a slightly (0.02%) higher (read: worse) average cost, in the long run we’d lose money.  Standardizing for the sake of standardizing hurt us.  And even though we settled on what we thought was a good standard, it was not good enough to be worth doing.  We’d actually lose more money by controlling the process than by letting it run wild.

So there you have it, standards for the sake of standards is not a universally good idea.  Sometimes chaos turns out a better result than controlling it.

The decision making drop off

I was noticing the other day that there’s a drastic drop off to determine if a person will take action.  Over the past few weeks we’ve been in the process of urging people to either fill out a survey or sign up for certain mailings.  To measure how useful the outbound communication was, we look at the rate of activity before the sign up and for the next few days after until the activity drops off.

What I found was pretty consistent.  If someone is going to respond to a request, it usually happens within 2-3 days or not at all.   Beyond that point all you seem to get are a rare handful of responses.  I’d speculate that those that do respond so late were probably not in the office at the time of the request and so responded promptly when they returned.

I’m interested by this behavior for a couple reasons.  One, it gives you a good idea about how often you need to communicate in order to get a majority of people to do something.  In the first 2 days you’ll get a chunk of people, and then you pretty much know what you’re going to get.  If you need more people to respond, you’ll need to communicate again.

Secondly, it’s an indicator of how long we really need to leave surveys open.  I haven’t tried it, but I wonder if we reduced the window of time that people were allowed to respond from a couple weeks down to 3 or 4 days if the response rate would change.  Is the amount of response reduced by a lack of urgency to respond? Is someone out there saying “eh, I’ve got two weeks, I’ll get around to it.”  If you cut the time down, would more people react, or do people decide in an instant whether they’re going to act or not?  If so, I think you could still cut the response window to 3-4 days and stop waiting around for your feedback.

So there’s an easy way to cut one to one and a half weeks from any data collection effort.  All that waiting time after a few days isn’t getting you anything anyway.