Over-Engineering #2

After a long day of holiday shopping, we decided to order Chinese food for dinner.  I picked it up on the way home and when I got home I unpacked the bag.  Included in the bag was this:

It’s soy sauce.  A small amount, maybe an ounce or two, in a little plastic container.  And then the plastic container is inside a little plastic bag which is tied tightly.

It’s clear why they did this.  Soy sauce in little plastic containers with lids that don’t always stay on is very prone to spilling and making a big mess.  The plastic bag is like the second hull on an oil tanker.  Breach the first and you have a backup system.

It’s also insanely stupid.  OK, I’ll grant you that buying soy sauce in bulk is probably cheaper than buying those little 1/2 ounce packages people get.  But, there’s a cost associated with packaging soy sauce by hand.  It costs you labor to fill the containers.  It is labor to cap each container and it is labor to then wrap each container in a little plastic bag and tie a knot in the bag.

So, my questions:  if your bag of Chinese food suffers enough damage to break open the inner container, how often does the outer bag get ripped as well?  Is the knot water-er, soy sauce- tight or will it just leak out there instead?  Is the savings in bulk soy sauce really still there once you pay for little plastic containers, plastic bags and the labor to assemble these silly little packages?

Some options for our Chinese restaurant:

1. Just buy the little packets of soy sauce
2. Find a container that won’t break open and package it in that instead
3. Maybe JUST use the bag.  If it’s sturdy enough, don’t bother wasting money on the little plastic cup

When all else fails, communicate…

Sure, I’ve railed against communication as a solution to your process problems, and I stand by that position.  However, I have found an exception while sitting in the dark and cold.

In case you didn’t know, the northeastern United States was ravaged by a particularly nasty ice storm late last week.  I’ve never experienced anything like it in my life.  The lights went dark about 9pm on Thursday and then the noises began.  In the darkness outside, between the patter of freezing rain were horrific creaks and cracking noises followed by deafening crashes as ice snapped trees like toothpicks.

We live under a canopy of trees close to our house.  It is probably the closest I’ll ever come to being in a war.  I lay in bed, wide awake, still and quietly listening to a constant chorus of destruction.  I didn’t get a wink of sleep.  In the morning, I opened my front door and witnessed the destruction first hand.

Not knowing what else to do, and having been told after calling the local power company that it could be a week before we’d have power back, my family and I packed up and left for greener pastures – namely, family who had power.

For two days, I called my neighbors and the town looking for signs that power would be restored sooner rather than later.  By Sunday, I had found a generator near where we were staying and left my family behind to return home and care for our house. 

Each day, sometimes twice a day, I would call the power company and ask for updates on our street.  Every time I was met with useless answers like “we suggest you make alternate plans” or “everyone will have power by the end of the week.”  They might as well have told me to go hell.  They were useless.

On Tuesday morning, the National Guard showed up on our street to clear it.  By noon we had a crew on the street repairing downed lines.  I don’t live on a very long street, so it took them all of half an hour to get everything back where it belonged. 

Despite how much overtime those crews were putting in, when I went out to talk with them they were friendly, cheery and helpful.  We were told we’d have power back in an hour or two.  Then they left.   That was at 1pm.  I gave them the benefit of the doubt, but by 5pm, with the sun gone and a cold night setting in, I decided I wanted answers.  Since calling the power company yielded nothing, I decided to go to their office.  We have a local light department, so the office is right in town.

When I walked in, the place was abuzz with activity.  A steady stream of people came in and out asking about their streets.  Everyone was met with unspecific answers.  Nobody seemed to know anything.

Frustrated, I left and called the town manager.  I got his assistant, who while very nice, essentially informed me that even the town manager didn’t know what was going on.  There appeared to be no plan, no list of which streets were being worked on now and which would be worked on next. 

I had enough.  Sure, it’s useless, but I called back the light department and started yelling.  I wanted answers.  I wanted to know who had the answers.  Someone had to know something about my street!  I got another useless secretary on the phone.  I demanded to talk to the general foreman.  I was summarily denied.  I told her my phone number and name and said to have him call me with an answer.  I never got a call.

I emailed the town manager, the general foreman and the assistant town manager demanding someone respond with a clear straightforward answer as to what was happening, or not happening, with my street.  Silence.

Sometime during that night, the power came back on.  Yet nobody seemed to know that it would happen.  Nobody was able to say to me that power was just minutes or hours away.  I eventually found out because I had started a practice of randomly calling my house to see if the answering machine would pick up.  Sometime before 9:30pm that same night that I was totally stonewalled, I got power back.

I was glad to return home, but I was furious with the process.  There were so many things that could have been done differently.

First off, I have no complaint with the speed at which power was returned.  I do not blame the crews on the ground for anything.  With as much damage as there was, I understand almost 200 people were in town just trying to get power back.  I blame management.

There’s a process to restoring power to a street.  I haven’t the foggiest clue what it is.  But it’s that kind of information that would be useful.  Expectation setting is important. 

Correction.  GOOD expectation setting is important.  I was told “end of the week” but that was unlikely.  No, I don’t think it would take longer.  In fact, it was highly probably it would take less time.  Consider the situation.  A few major power lines distribute power to the rest of the secondary lines.  Then individual lines from the street to your house connect you to the grid.  Fix the major lines, restore a lot of power quickly.  The distribution is much more likely a heavily right skewed curve. 

Most people get fixed in the first few days because you go for the biggest bang for your buck when recovering from a disaster.  Then, you start dealing with the oddballs – the excessive damage to a single home, etc.  Sure, there are people who won’t get power until the end of the week, but the chances you will get power before then are much greater.

The only way end of the week would have been a reasonable expectation to set would be if it was heavily left skewed.  For example, if the process required that every downed line between the street and someone’s house had to be fixed before anyone got power, then most people wouldn’t get power until all the repairs are done.  But that simply wasn’t the case.

So, what could the town have done differently?

1.  Estimate.  You have to know how many downed lines and trees and etc you have in order to know how much help to request.  Someone must have surveyed the damage while crews got to work.  What’s the mean and standard deviation for fixing a downed line, broken transformer, etc.  Figure out that and you can estimate how much effort to fix a street.  Know that and you can prioritize the streets and start estimating when you’ll reach various streets.

2.  Inform.  What’s the process for fixing a street?  Be specific.  First a cleaning crew shows up.  Then one or more linesmen reconnect the lines.  Then what?  Had I known what happened next I might have understood why it took until late that night to get power back.  Maybe there’s a system imposed waiting period for safety reasons.  Maybe inspections are necessary.  Maybe someone just has to flip a switch.  I have no idea!  But someone knew, and if they shared it, and expectation would have been set!

3.  Provide relevant information.  The town website only said how much of the town had power.  Guess who isn’t looking at the town website – people with power!  Duh!  People with power don’t care anymore.  People without power don’t care that 55% of the town has power!  They care about when they are going to get power!

4.  Leverage appropriate channels.  Guess what?  Old people, and my town has lots of em, don’t use the Internet.  They’re not getting the input because they don’t know to look for it.  Thus, you get overwhelmed with phone calls because nobody even knows you are communicating anything at all.  Even if isn’t particularly useful.

Here’s the thing.  The town couldn’t control the speed at which repairs are made.  Ok, they probably could, but I’m not even going to examine how possibly poor their repair process is.  God knows if they even had a disaster plan.  It sure didn’t seem like it.  But let’s assume they did and let’s assume it was relatively optimized.

They had a lot of work to do.  It was going to take a lot of time.  If you can’t change that, then you need to communicate more than “end of the week.”  That’s not communication and that doesn’t help people whose lives are affected plan what they should do next.

The amount of time without power might determine: Do I pay for an extended stay in a hotel?  Do I buy a generator?  Do I just shiver?  Will I have to discard the contents of my fridge?  There are financial implications!

When disaster strikes, whatever process you have to deal with it may be adequate, but that’s hard to tell from the outside looking in.  If you’ve just got too much to deal with, communicate.  And good communication depends on a) knowing what your process is and b) knowing how your process performs so that you can set realistic expectations of the result.

But we’ve always done it that way

It’s another example of thinking that what you have is what you need.  This time it’s a story about priority and severity.

What the heck are priority and severity anyway and why should you care?  Well, if you’re a quality assurance person, apparently they’re both very important.  Priority is the order in which you as a tester want things fixed.  Severity is “if this defect made it to production without being fixed, how bad would it be.”  There is a difference in theory.  For example, let’s take the case of legal disclaimers. 

As a QA person, even if the legal disclaimer is wrong, it doesn’t block you from testing anything.  It’s just text on the screen.  So, as a tester that would have low priority for you.  However, if it went to production with inaccurate legal content the ramifications for your business could be quite dire indeed.

With that justification in mind, one could argue that priority and severity have very important and separate purposes.  Well, that is until you look at the data.  For some reason, probably just experience or insight or whatever, a senior manager said “do we really need both fields?”

Sure enough, the managers of the team jumped to the defense of the two fields.  Oh yes, they exclaimed, this is how we manage our tests.  Anyway, I pulled some data.  Some is probably an understatement, actually about 10,000 defects that had been logged over the past years.

And then we brainstormed some questions that we wanted answered.  Is priority or severity affecting how quickly bugs get fixed?  Do priority and severity move independently of each other?  Is priority or severity being downgraded in order to get things into production?

The answers were fascinating.   For one, neither priority nor severity affects either the median or interquartile range of duration it takes to fix a defect.  To clarify, if you rank something as a high severity item it takes about 5 days to get it fixed.  If you rank something as a normal severity, it takes about 5 days to get it fixed.  And the same goes for priority.  With one exception.  If you rank something as low priority it still takes a median of 5 days to get it fixed, but the interquartile range changes.  Instead of being from 1-10 days it becomes 1-30 days.

Secondly, we checked to see if priority or severity move independently.  We have two attribute variables – priority and severity.  Priority can be 1, 2, 3 or 4.  Severity can be 1, 2, 3, 4, 5 or 6.  It’s a simple test to do a chi-square table.  And what do you know, statistically significant evidence that priority predicts severity…

Finally, is priority or severity being changed in order to get things into production.  Honestly, this was a bit of tangent, but someone brought it up in defending the use of two values.  They pointed out that these values were so important that people would change them lest the release management gods find out that a high priority defect wasn’t fixed before the install.  It’s not true.  Of the 10,000+ defects we looked at, less than 2% ever had their severity revised at all.  Of those 2%, about 33% had their severity increased while the 66% had their severity decreased.  Yet, for the ones that did have the severity decreased, there was no evidence that the changes occurred in conjunction with install dates.  In fact, changes in severity were evenly distributed throughout the month.

The point is this.  All these people thought that having two values was an advantage to them.  They needed two values to get their jobs done, they claimed!  I admit, it’s a small victory for data analysis, but it points out something important.  Just because you’ve always done it that way doesn’t mean it’s right or sensible or even that you’re getting anything out of it.  Going with your gut, not such a wise choice.

By the way, it’s a good change management story since this new senior manager really likes data.  And convincing the most senior person that you’ve done your homework is an easy way to get the troops to fall in line.

It isn’t what you have, but what you want…

I think it’s a Sheryl Crow song that goes something like “it’s not having what you want, it’s wanting what you’ve got.”  I like the song, but the sentiment is downright incorrect.  It is, in fact, at least when it comes to process, all about what you or, more importantly, your customers want.

Let me rewind.  I was in a meeting, perhaps a couple months or more ago, where we were asked to come up with a new set of key performance indicators for the team.  KPIs are a very powerful tool when it comes to process improvement.  I can choose KPIs that incite good behavior or ones that create behaviors that I don’t want.

As an example, if you measure your developer’s productivity in lines of code, you won’t necessarily get more functionality from them, but you will definitely get more lines of code.  It can actually cause people to write sloppier code in order to game the metric system you put into place.

Anyway, being that KPIs are an important thing, I thought the right place to start was “what do our customers want from us?”  Apparently I was wrong, because when I brought this up I was told “no, let’s start by looking at what we do today.”

What!?!?  Who cares what we do today?  Well, let me rephrase that… DON’T CARE WHAT WE DO TODAY!  Today isn’t working.  Today, we get results which our customers are unhappy with.  Today, our KPIs aren’t aligned with our customer’s needs.  Sure, we’ve got stuff, all kinds of stuff that we measure.  It’s stuff that doesn’t matter!

People forget that just because you have it doesn’t mean it has value.  Just as you lean out your processes, you should be leaning out your measurements to the few things that matter.  Everything else, forget about it.  Who cares if you knew that information in the past.  If it has no impact on whether your customers are happy, stop looking at it.  For god’s sake get some focus!  Specifically, get some focus on what your customers want from you.  All the rest, even if you measured it before, should find its way to the trashcan.

Process Platitudes

I just got out of two days of training at work.  It was on a topic I wasn’t particularly interested in, but part of the training had to do with dealing with difficult people.  While on that topic the trainer proudly wrote “E + R = 0″ on the board.  He then turns to us and says “Event plus Response equals Outcome”.

Pretty obvious, huh?  Something happens and how you react to it will create either an outcome you want or one you don’t want.  It’s a true statement.  It’s a true and USELESS statement.  It’s a platitude, a well known idea that you can’t take action on.  I didn’t come to training for this!  

E+R=O is a process.  In this case, it is the process for dealing with an interaction.  It’s a challenging process because the situations arise randomly and if you understood the key factors you could probably effectively choose from one of several responses and diffuse the situation.  The challenge is that you have to think on your feet.  But, if you could freeze time, you could analyze a situation like this as you would any other and arrive at a good response.  That is, if you had the details of what to do in which situations.  E+R=O doesn’t give you that, of course.

I might as well have said to all the people in the process “I plus P equals O.  The inputs plus the process make the output.”  No kidding.  Yes, it’s true, but I can’t give that as advice to someone in a failing process and get better results.  It’s not like some light is going to come on when I say that.

“Oh my god! Why didn’t I think of that before?  If I just stopped screwing up the process everything would come out OK!” shouts Bob.  Suddenly all closed doors are opened to him and a great weight has been lifted from his shoulders.  The revelation that he could just stop screwing up never came to him before.

Process fixed.  I go home a hero.  Yeah, right.

USELESS!  I can’t suddenly just stop screwing up the process or I would, wouldn’t I?  So there’s no point in espousing such nonsense expecting to change the world.  In improving a process, what good would it do?  When people start giving out strategies for completing a process, you’re at the same level.  Processes thrive on actionable behaviors.  Stay at the high level and what action can I take?

Responsibility and the process

Over at the Evolving Excellence blog, while catching up from a vacation I read this blog entry about the abdication of responsibility.  Here’s an example of one of the most highly paid leaders not taking responsibility and perhaps even being in denial of his part in the downfall of Citigroup.

It got me to thinking about what someone’s responsibility is within a process.  If you are a smart individual and I set you to a task, I have the expectation that you’ll complete it.  If I give you no direction on how to get it done, and it doesn’t come out right, who is at fault? 

Well, the person, to some degree, for not being capable of completing the task and myself, to some degree, for a host of sins including possibly selecting the wrong person for the job, providing inadequate direction, etc.

But what if I had given you a process to get it done?  What if you tried to follow the process in earnest and still produced a defective product?  Would it be right of me to blame you for screwing it up?  I think not.  There is a responsibility of every employee to do the process you have laid out as best as possible.  But beyond that, I have a hard time assigning much blame to the employee for defects.

Why?  Well, in the cases where I have designed a development process, I never allow an employee to work in a vacuum.  Typically, to assure the quality of work, every employee’s work is checked by his or her peer.  If both people believe that the analysis is right, the design is correct and the coding is faithful, why should just the author or programmer bear the brunt of a failure?

On the other hand, if the employee chooses not to follow the process, well… just as all bets are off as to how the result will turn out, so is any guess at how I might react to a defect.  In the cases I’ve encountered where I can clearly see the process has not been followed, I can assure my reaction has not been to shrug and say “try better next time.”

In fact, I have told my employees that the process shields you from blame.  If you follow the process properly and in earnest and something goes wrong, as the owner of the process, I will shoulder the blame.  However, if you decide to strike out on your own, and you screw it up, then the blame is yours to bear. 

I’m not suggesting that employees are responsibility free, but if you replace free thinking with structure because you believe it will produce a more consistent or better result, you can’t then flog your employees for your bad process design.

As for the story which inspired this post… since the top executives don’t follow a process, you can’t excuse their behavior.  In fact, since the way they decide to run the company likely forms the foundation for the processes that will get used, the blame belongs squarely with them.