Why errorproof when you can double-check?

OK, so riddle me this… if you were a programmer and were writing some software would you:

A) make it possible for the user to screw up using the software and then spit out annoying error messages after the fact?

OR

B) make it so that the user did the right thing in the first place?

Please tell me you answered B.  I know you didn’t.  I wish you did, but I know you didn’t.  Why?  I have no idea why.  That’s what makes it a rant!

Just today I was talking with a group of folks who had developed a template.  It was a generally good template, but they sent it along with another tool that checks the template to see if users have filled it in properly.  So, let me get this straight… we coded a template that the users are ALLOWED to screw up.  Then we coded another tool to check to see if the users screwed it up!?!?

If you are in the position to prevent the error in the first place, why wouldn’t you?  And, I’d argue, if you can write a tool to detect the screw up – ie, it is possible to programmatically figure out that the template is wrong, so it requires no great amount of human intelligence - then you MUST understand enough about the system that you could have coded it in such a manner that the users didn’t screw it up in the first place.

Call it front end edits, call it whatever, but why in gods name would you allow your user to do a bad thing and then punish them later for it?  It just doesn’t make any sense!

I get that some things are hard to error proof.  Manual processes are hard to errorproof.  People want to do the wrong thing for some crazy reason.  They like subverting processes.  But, why, oh why, would you allow the same thing in software?  It is one place where you can completely block a user from doing the wrong thing.  And yet… you develop a template and a governance tool to make sure people are using the template right?  Just make it so the template gets used right and throw away the stupid governance garbage!

Let’s try it my way

I’m not exactly sure what to file this under other than “just plain frustrating.”  Dr. Deming would have called changing a process without understanding it tampering.  This rant might apply to my green belts, to my sponsors, to almost anyone who doesn’t understand the value of understanding.

How sick am I of hearing about solutions to problems when the cause of the problem is not understood.  “It’s a training issue”, “it’s a resource experience issue”, “let’s standardize that process.”  I’ve reached the end of my rope.

Myth 1:  We have hired particularly intelligent or skilled workers and therefore the fault with the vendors we work with is that they have not.

Reality:  Nonsense.  Every company wants to believe that they’ve hired the top talent.  Define top talent?  The top .03% of the population?  Hellooooo… all companies can’t have employed the top .03%.  There aren’t that many people to go around.  Odds are really good you’ve employed pretty average people; THE STATISTICS ARE AGAINST YOU!  Think the Toyota philosophy: really great processes + mediocre people = good results.

Myth 2:  When in doubt, training will solve everything.

Reality:  Training solves very little.  People quit.  People forget.  People ignore you!  People subvert the process because they can.  Training implies people are well intentioned.  Forget training, just make it easier to do the right thing than the wrong thing – errorproof!

Myth 3:  Standarized process will solve the problem.

Reality:  What standard?  Any standard?  Is a bad standard better than no standard at all?  Could you replace good judgment which produces variable results around an acceptable mean with no judgement that produces a very tight spread around an unacceptable mean.  You bet you can!  At least you’ll fail in a standard way!

And yet, we march forward relying on the same time-tested bad advice.  Why?  In the absence of any data or the thought to get any data, people learned to go on their guts.  Now that they get data, they look at the output and say “oh, that’s no good.” and go back to their gut for the solution to the problem.  Well guess what, if you had made more good decisions than bad decisions, you never would have hired a six sigma person in the first place.  Things would be working out for you instead of where they are today.

Let’s try it my way and think “my gut is wrong, I will believe the data” instead.

I don’t believe the data

What do you do when the data you gather doesn’t meet the expectations of the audience you have to tell?  One of the Green Belts (GBs) that I’m mentoring just gave a presentation on the first go-round of a measure phase.

My poor GB was forced into doing a baseline measurement because, while people felt there was a problem, we had no idea how big or small the problem really was.  I want to give my GB credit for doing her homework the right way.  She got statistically significant sample sizes.  She properly randomly (not just haphazardly) selected samples from the population.  She conducted careful time studies with the resources who actually do the work in question.

And when we presented the data to the sponsor, he concluded that the data was suspect.  Why?  Because he had a different expectation than the data showed!  My GB was collecting data regarding the effort that goes into testing.  While trying to figure out what the financial opportunity might be, we needed to know how much effort went into writing a test case.  How would you time it?  We got a stopwatch and had the test case writer record the start and stop times for each test case written.  We didn’t even tell him what the data was for, so he had no clue whether a higher or lower test case writing time was preferred.  Hopefully that at least eliminated deliberate bias, though I suppose the writer may have assumed one way or the other was better, but I doubt it.

What’d we find?  The 95% confidence interval for median test case writing time was between 201 and 261 seconds.  Note I said median time, so you know that my GB did the right thing and checked to see which statistic (median or mean) was more descriptive for the data collected. 

What’d the sponsor say?  “The data is suspect, test case writing should take at least 10 minutes per case, I’d expect.”  WHAT!?!?!  Where’d he get that number from?  Gathering data isn’t about matching someone’s expectations; data is about confirming reality.  Whether the sponsor liked it or not, the data showed that’s how long it took to write a test case by these resources.

So I found myself saying to my GB after the meeting, “just ignore them and trust the data.”

One might conclude that taking so little time to write a test case is the problem, but it’s not like you can just say to your resource “take longer to write a test case.”  Well, you can, but you can be sure that the resource will still take somewhere between 201-261 seconds to write the median test case and spend the remainder of the time twiddling his/her thumbs.  The time taken to write a test case is dependent upon the effort put into it.  Change the nature of the effort, change the time it takes.  But you can’t change the time directly.  All other things being equal, at a speedy 201-261 seconds per test case, why would you want it to take 10 minutes per case instead?  If you said “that’s too long” I could see a complaint, but “that’s too short”?  Baffling!

Negotiating a Change

Today was an interesting day in change management.  In my job, I enjoy a certain neutrality of my position.  Since I don’t develop software or test the software, I have no vested interest in either party’s position on an issue.  As an outsider looking in, it’s amazing to me how people can turn an easy conversation into a really painful one with a minimum of effort.  This particular battle royale was between the Quality Assurance organization and a Tools Development team.

Let me start of by saying, both groups are in the wrong.  For one, the tools development team developed a tool for the QA department to use.  How they accomplished this I’ll never know, since they failed to actually involve their customer, QA, in the requirements work.  Bad idea number one.

As for QA, under most circumstances they’d be in the right to refuse delivery of the product, but this isn’t a free market.  It’s a company.  Clearly some senior leader decided that QA needed said tool or it wouldn’t have been built in the first place.  QA was not in a position to flatly refuse.  They were justified in asking for some changes.  How they asked for the changes, though, was bad idea number two.

So here’s a play by play of the entrenchment of both sides:

QA representative:  “We have some issues with the template we use to load the data into the tool.  We need X, Y and Z done to it.”

Tools team representative:  “I don’t understand why you need X, with our tool, X is pointless.”

Repeat point A and point B ad nauseum until I interrupt.  As each team argues his or her side, the other team digs in.

This is as close to verbatim as I’m willing to get without worrying that the two people I’m quoting above might read my blog and say “hey, that’s me!”  Nevermind the frustration that I’m feeling because I prepped said QA representative for how the conversation should have gone.  What’s the point of preparing someone if they don’t listen?

Let’s pick apart the QA rep’s word choice.  Words like “issues” and “need” have no place in the start of a negotiation.  Absolutes are not negotiating.  If I say my car is for sale for $10,000 firm, that means I am not discussing a lower price.  Negotiation is over before it could begin.  You could literally see the members of the tools team just clench up at this statement.  The Tools team rep’s words were really no better, but I didn’t prep them for the conversation so no surprise there.  “I don’t understand why you need…” says “I’m smarter than you and have already solved your problem.”  Of course, this offends the customer since it is clearly untrue.

Here’s how the the conversation should have gone:

QA rep:  “We’re encountering some resistance helping you guys roll out the tool.  I got a chance to meet with some of my folks to understand what their concerns are and I think we can smooth things over by helping them to put their fingerprint on the template.”

There’s a lot different about this statement, obviously.  First off, the QA rep indicates that s/he is interested and willing to help the tools team roll out the tool – “I’m on your side.”  From there, “I met with some of my folks” indicates, “it’s not me who has an issue, I’m here advocating for a bunch of people, however foolish they may be.”  And finally, “helping them to put their fingerprint on the template” says “here are some changes we’d like to see made.”  Look closely and you’ll understand that this is the exact same statement that our actual QA rep made very badly.  This statement has the same goal, but this statement avoids the entrenchment in “our” position.

How do I know this second statement would work?  I waited until the teams were done arguing and the meeting was over and I sat down with the tools team manager and said exactly that!  And guess what, despite the disastrous first hour meeting, I got the exact same thing our QA rep wanted but without an argument.

And what did I give up for it?  Well, I had to put my tail between my legs and ask them for “help.”  Do I like doing it?  No, it always makes me feel a little dirty, but it works very reliably.  This goes way back to my very early post about appealing to someone’s ego.  Being deferential to a person who has something you want puts them in an artificial position of power.  People like being in control and using their largess to help poor me out.  They think “there’s nothing wrong with the tool I created, but these poor stupid QA people have no clue and if I must come down to their level and speak in small words, so be it.”  And indeed, what QA was asking for were minor concessions in the big scheme of things.  So it wasn’t a complete lie.

I’m continually amazed how people think that they can come in with anything at all, and because his boss or her boss said to do it that they believe it now carries sufficient weight that it will simply occur. People fail time and time again to recognize that if you both don’t report to the same boss (and even if you do) that you can’t expect people to think your ideas are a priority.  Get over it and learn how to work someone else’s ego to meet your needs.

“Diplomacy is the art of letting someone else have it your way.” – Daniele Vare.

Annoying does not equal problem

Sadly, I had to tell one of my Green Belts today that she did not have a project to work on.  When we kicked off her project, she was looking at a quality issue with our testing resources.  Impressions were that our resources were terribly inefficient at creating test cases and that the rework costs to redo what they screwed up were costing us a fortune.

We didn’t have data to support this hypothesis, so our first goal was to baseline the performance of the testing team.  She did a great job collecting the rework numbers from a random sample of teams.  The sad reality was that the impression did not line up with reality.

The 95% confidence interval for waste was between 0 and 80 test cases per project.  The thing is, it only costs about 3 1/2 minutes to write a test case, so even in the worst of all cases you’re talking about 4-5 hours of waste per project.  This is definitely not worth pursuing.  There are simply bigger issues that she could be working on.

But the point of telling you all of this is not to tell you about a project that didn’t pan out.  It is to tell you about the way people identify problems they want to work on.  Despite the very low cost of rework being incurred by the team, this particular project was very high on their list of things they wanted fixed.  Why?

My guess, because the problem annoys people.  How frustrating is it that you have to redo something that someone else does?  It’s really annoying, I can tell you.  I hate spending my time fixing someone else’s screw up.  After all, what am I paying someone for if I have to deal with it anyway?

What people don’t think about, and should be concerning themselves with, is not what’s necessarily annoying about their job, but what is their job in the first place?  Doesn’t it baffle you that (in the best interest of the company) people aren’t saying “hey, wait a minute, the only reason I’m testing in the first place is because the developers screwed up the code in the first place!  Maybe we should help the developers write better code.”  It is impossible for someone to think outside the boundaries of the job they were given and really understand if they should be doing their job at all.

It’s probably a conflict of interest for most people to think this way.  Who wants to admit they do something that doesn’t have to be done?  Everyone justifies themselves as being important to the operation of the company.  Convinced that our job is a critical component, we focus improvements on the stuff that annoys us about the job we think we have to do.  Annoying as they may be, it’s hard for these irritations to add up to much in the way of an efficiency gain.  Maybe some jobs shouldn’t have annoying parts simply because they shouldn’t be jobs at all.

Your everyday, “average” disaster

Here’s an interesting real life example of not getting the basics right.  There was a project that I was looking at the data for just the other day.  It was a LEAN project, so the black belt assigned was hoping to reduce the cost of resolving defects (our “Y” in this story).  The BB, let’s call him Bob, did a nice XY Matrix to brainstorm the things he should measure.  Bob even did a nice job collecting the data; he carefully conducted a study to produce a value stream map.

And then Bob promptly screwed up his entire project!  Bob had a few potential X’s for his project.  He made the same mistake with all of them, so I’ll just focus on one of them.  The X in question was the number of rejections per ticket.  Now, I can’t say that Bob did a correlation test between this potential X and Y – which may have been his first mistake.

His second mistake was not running a normality test on the data.  How someone forgets to do something so basic, I’ll never understand!  If you don’t think normality matters all that much, let me tell you how the story played out as a result of this decision.

So, Bob assumes the data is normal and calculates the average rejections per ticket.  The number comes in at around 5 rejections per ticket.  “Five rejections per ticket?, ” he thinks, ” this is clearly a problem and we can fix this.”  Bob implements some improvements to help assure that good information gets into the ticket in the first place.  He re-measures after the improvement is in, and wouldn’t you know it, the average number of rejections drops to 2.5.  “Success!, ” Bob declares, ” now that the average rejection rate is down the cost of resolving defects will be greatly reduced.”  I’m making up Bob’s dialogue because I wasn’t there with him when he made this errant claim and I imagine him saying silly, badly-scripted things with a false sense of confidence.  Kind of like the Bob from those enzyte commercials…

Except that he’s wrong… see, the data wasn’t normal.  The high average number of ticket rejections is due to a handful of tickets each month that are so sloppily entered that they get rejected dozens of times before they are correct.  For most tickets, the number of rejections is between zero and one, but those very few disaster tickets skew the average to be high above the more descriptive statistic – the median!

Bob’s improves did improve something.  The improvement eliminated some of the worst offenders in ticket rejection, but since the vast majority of tickets didn’t have this kind of rejection rate, changing the X failed to have any effect on the Y.  Congratulations on failing, Bob!  Next time pay attention in Black Belt class.

I think after someone claiming to have expertise makes this kind of amateur screwup that we should be allowed to send him or her to remedial statistics.  I can almost understand why someone might not review the residual plots for a linear regression, but not running a normality test?  Foolish, at best.

Not invented here, not used there

So here’s an interesting conundrum. Let’s say that I’m tasked with standardizing some processes for my team (never mind my rant about the appropriate scope of process standardization) and let’s say that another team I know already has a fully formed set of processes I could adopt.

It seems to me, assuming that it is otherwise appropriate to do so, that I should take this other team’s processes and simply not reinvent the wheel right? Well what if it turns out that the other team has a completely specified process that they don’t use?

Yeah, ok, so my first question is why did they write down a process they don’t execute? Obviously, this is an adoption issue. Some smart people (and I knew these people and definitely think they’re genuinely smart) decided to set up a standard process. This is a good thing. But, they lacked the change management capabilities (whether it’s a really good sales pitch or adequate command and control) to make it a reality.

On my end of things, how do I make an objective evaluation about this process? The most logical option when deciding whether to adopt a process would be to look at that processes’ capability to deliver results that meet your CTQs. But a process that is not executed has no data to support its goodness.

Unable to put my hands on any data that says the process is good, there’s but one course of action in my mind – dump it into the “not invented here” pile and ignore it.

Wouldn’t it just be great if we could take completed processes, like CMM or Agile or RUP or whatever and just use them with success? Ever notice how all these processes come with a complete lack of supporting data about their capability to deliver results? There’s always the big end result “50% decrease in time to market” or “66% increase in defect containment” or whatever, but they never actually talk about the correlation between the result and the process activities. People wanting you to adopt a process are selling something, so if the story is totally rosy one can bet that some data has been conveniently omitted.

“Not invented here” is not a great policy, but I’m sure that “invented (but unproven) there” does not make a good enough case to start using someone else’s stuff.

By the way, when I set out to start writing this blog entry I swore I was going to tell you the exact opposite – that a good idea even in the absence of execution is still valid. C’est la vie. If I can’t convince myself of that, I certainly won’t be able to convince you.

A disturbing lack of integrity

Of all the skills that a Six Sigma person needs to be successful, I’d say two things stand out.  The first is the ability to analyze data for statistically (and practically) significant patterns.  The other skill is change management.

So, when I observed a person using the first skill to manipulate the data to enable the second skill I saw a person treading a very dangerous line.  A healthy foundation of change management is the trust that you have built with the folks you are trying to influence.  You use data to make your case, build trust in your analysis and show that you are not being manipulative.  And sadly, as I’ve just observed, sometimes (mis)use that trust to hide the data that doesn’t fit your case.

Where I work, Six Sigma is still a fairly new concept.  If an individual in the Six Sigma organization misuses the trust we’ve built up, s/he can do more damage to that trust in a single act of subterfuge than we can hope to rebuild with months or years of honest, forthcoming behavior. Lack of integrity on the part of even one individual in your organization has consequences that reach far beyond the individual.

Here’s what I saw today.  As a team, we were working towards a presentation to a group of senior managers.  Each project manager was going to have to present his/her project, goals and results on a one-page slide and then speak to it for just a few minutes.  This was purely information sharing.

One of the charts showed a stunning rise in First Pass Yield of a given type of transaction.  It looked kind of like this:

What a great story!  First Pass Yield from the 30-40% range to 70%!  Who wouldn’t be excited?  Well, who wouldn’t be excited except that this chart is truncated.  Here’s what the data looked like a few months after May…

In June through September, the numbers were better than they were in January through April, but only marginally so.  I haven’t calculated control limits, I doubt excepting for May that it’d even show special cause variation.  It wasn’t nearly the great success story.  Now, one might chalk this oversight up to optimism.  We could have simply claimed success too soon in May.  But we weren’t preparing this presentation in May of 2007.  We were preparing it today, nearly one year later!  We knew that the gains in May didn’t hold; we had the data in our hands which showed it.  And yet, one of my peers was prepared to show a slide which conveniently cut off the data before it went south.

What’s worse, is that we make all our data, including this disastrous collapse in first pass yield, available online.  It wouldn’t take much for one marginally smart individual to connect the two and say “hey, that isn’t what you showed us in that presentation!”  And, if that realization gets back to the right (or wrong, as it were) person(s) then our entire group’s credibility is damaged.

It may be convenient to lie now because it gets you your Black Belt certification, but these kinds of actions serve only to undermine the longevity of not just your, but all our careers.  Change Management, and the required trust to accomplish it, cannot be sacrificed for short term gains or personal reward.  Once the trust is undone, you can be sure that nothing else you try will ever make a difference because nobody will allow you to implement a change again.

It takes an enormous amount of integrity to do the right thing, but it is better to admit that more has to be done to meet your customer’s CTQ than to sweep it under the rug.  Your lack of integrity will catch up with you.

Not using run charts

Run charts are a great thing, don’t get me wrong, but I’m starting to wonder if they have as much of a place in the monitoring of software development processes.

It stems from two issues:

1. We don’t produce a lot of widgets
2. The widgets we do produce take varying lengths of time to produce.

So we’ll pretend I’ve got the following projects, numbered 1 through 6.

Let’s say I make a process change in Q3 of 2007 to improve the requirements process.  My research indicates that it should result in a reduction of defects found in production at the end of each project.  Since requirements happens very early on in the project lifecycle, projects 1, 2, 3, and most likely 4 will have operated under the old requirements process.  If I were using a run chart to display defects found in production, it’d take until the end of Q2 2008 before all the “old process” projects made it out into production.  If I lumped all the production defects in a run chart, the old process projects would continue to skew the production incident data for at least another month or two beyond their install dates.  For a period of time, the old process projects could create sufficient noise so that I could not detect a difference.

So, I could separate the projects out into two pools – old process vs. new process run charts.  But the new process only has two projects under its belt by the end of Q3 2008.  Data would be so sparse that it’s meaningless. 

Displaying process data by months when the process doesn’t fit nicely into a fixed period just doesn’t make sense to me.  Even more so when it comes to the Xs.  If I’m making hundreds or thousands of widgets, and a few go bad because an X moves out of control resulting in defectives, I can take countermeasures and save the vast majority of the widgets.  If an X moves out of control for a project and it isn’t recognized until the run chart is calculated some number of weeks later, you could be out hundreds of thousands or millions of dollars trying to correct it.

Process management in a software development world collides with project management.  If we understand what process variation will result in projects going badly, then we should observe and act on a project by project basis, not some arbitrary monthly aggregation in a run chart.

I’m on the hunt to devise an MBF which can adequately represent individual projects that show characteristics of being in trouble.