A funny quirk of converting continuous data to attribute

Many a time I have had to explain to folks why they should not be converting data with more resolution into data with less resolution. On the spectrum of “resolution” you have attribute data (categories or characteristics you can assign to a given item), count data and continuous data. I’m going to assume, having refreshed your memory on the categories, that I needn’t explain them further.

One of the most common ones I see is the conversion of count data into attribute data on surveys. People will create a survey question like “how many days on average do you work from home in a week?” And rather than just leave a blank spot for someone to answer, they create arbitrary buckets like “2 or less”, “3-4″, and “5 or more.” I’ve always wondered, when someone does this, if they realize the impact of their choice. In the 2 or less category the person has managed to include 0, which to me is distinctly different from 1-2 times a week. On the upside, at least the upper boundary of this data is known – you can’t work from home more than 7 days a week.

Which brings me to my particular funny experience with attribute data. One of my Green Belts was working on a project and she wanted to develop a survey to collect some data. The question she wanted to ask was “what was the size of the project you were working on?” The answer she proposed had three buckets – small, medium or large. Her question to bunch of us was what’s the current definition of a small project, medium project or large project. Suddenly I realized that the problem here was that the “large” bucket had no possibility of an upper bound when converting continuous data.

For example, let’s arbitrarily define a small project to be between $0 – $250,000, a medium to be $250,001 – $500,000 and a large to be $500,001 – what? Suddenly the buckets don’t represent equally sized groups. Large has no upper boundary created by the next larger bucket. (On a little tangent, one thing that drives me crazy are groups that are $0 – 250,000 and 250,000-500,000. The upper boundary of the first category overlaps the lower boundary of the second category. So, if someone gave me a project that cost exactly 250,000 which bucket should I choose?)

If you are trying to make some determination about the nature of work being done using these small, medium and large characterizations, the large bucket contains everything up to and including the biggest project the company has ever done, but we stop making a distinction between large and very large, jumbo or humungous. And I wonder if the difference between small, medium and large mattered, shouldn’t the various scales of large matter as well.

At any rate, it’s a good reason not to convert continuous data to attribute data. Unless you know the true boundaries of the continuous data or you’ve discerned from samples of data the logical groups that naturally occur, you probably should just ask the question “how much did the project cost?” instead of giving them just a few buckets. From the small and medium buckets above, I’d assume that large was between $500,001 and $750,000.  A $10,000,000 project (and we do quite a few of those) would end up in the same group with the $750,000 projects.

If you’re concerned that the respondent might round off the answer, it’s really no worse than when you provide them arbitrarily sized buckets. You can always tell them in the survey that rounding to the closest $100,000 or whatever is fine or that exact answers are important. You lose a lot of information when the final bucket you have becomes a catch all for everything else.

Operational definition = stupider people

Some how Green Belt training has made our employees dumber.  No, this isn’t an example of Flowers for Algernon (it was a short story I remember from my childhood, go look it up), but something has gone awry.  Maybe ‘dumber’ isn’t a fair term, but the basic assessment is this: individuals who have not taken GB training are exposed to it and suddenly they forget what words mean.

In my specific example today, someone asked me what the operational definition of “unambiguous” is.  Occasionally, someone springs a twenty-five cent word on me, but unambiguous isn’t even a five cent word.  It has common use in our language.

So let’s get back to basics people and help you all out by defining “operational definition” in my own words.  An operational definition is an unambiguous (go look it up, if you’re confused about that word) description of how something will be measured.

For example, how do they measure “on time departure” for an airline?  Take a guess.  The minute the plane leaves the ground?  Nope.  The minute the plane taxis to the runway?  Nope.  The minute the doors close?  Nope.  It is considered “On time departure” when the emergency brake is released on or before the scheduled departure time.  Going to be running late by any rational person’s definition?  No problem, just release the parking brake and set it again.  You’ve departed “on time!” 

Operational definitions differ from dictionary definitions.  “Good” or “Bad” has a place in the dictionary, but neither word has a place in an operational definition because the words have multiple meanings.  Instead, regardless of whether the definition is perfect, an operational definition’s focus is on setting something up to be consistently measured regardless of who does the measuring.

Operational definitions assume you know what the words mean, for example “departure” in “on time departure” means to leave.  The issue isn’t that we don’t agree that departing means to leave, it’s how we measure exactly when someone is considered to have departed.

So, I’m convinced that Green Belt training has made people stupider.  Suddenly they’re not only misusing a concept from training, but they’re using it to ask questions about words they already know the meaning of.

Why my oil company can’t deliver

Well, my oil company can deliver and does so regularly – too regularly.  The problem is, if I didn’t have a whopping reserve fuel supply, I’d probably run out before they figured out it was time.  It’s not to say that the oil company doesn’t try to figure out my fuel use, but they’re stuck in a way of thinking that doesn’t accurately predict what I’ll consume.

In the oil industry, the determination to deliver oil is based on your home’s K-Factor.  The k-factor is a simple calculation for any given period – the number of elapsed degree days divided by the number of gallons of oil used in the period.  What’s a degree day, you ask?  A degree day is a measurement of how cold it is outside.  To figure out if your house needs heating, you take 65 degrees and subtract the average temperature for that day.  To calculate the average temperature, simply take the high temp plus the low temp and divide by two.  So, the formula for calculating one day’s degree day value is (65 – (( high + low ) / 2 ).  And thus, if 1000 degree days have elapsed and I’ve used 100 gallons of oil during that time, I’d have  K-Factor of 10.

The oil company, having measured degree days and oil use (because they know how much oil they delivered you) can now ostensibly predict when you’ll need more oil just by counting up the elapsed degree days.  For them, degree days / your k-factor = gallons of oil used.  The number gets low enough, they send out a truck for a fill-up.

I was curious as to how well this worked in reality, and I like collecting data about my energy usage.  I built a very simple linear regression with my predictor being degree days and my gallons of oil usage being my response.  Guess what, the prediction capability of using just degree days has an r-sqr of something like 10% for my house.  It’s an AWFUL prediction of my oil use.

And it’s easy to imagine why: how does it feel when it’s sunny out?  cloudy?  calm or windy?  raining or foggy?  I can think of lots of things that would appear to influence how much oil I’d use to heat my house.  Snow is a good insulator (at least the Eskimos seem to think so), I’d expect that would make my house more efficient to have a roof covered in snow.

Getting the extra data isn’t hard – NOAA offers up tons of weather data for free, so I went and got some more data.  I got data about precipitation, wind speed (both sustained and gust) and visibility (in my mind, a good measure of how cloudy it is) for every day since I’ve been collecting usage data on my oil consumption. 

I ran another regression analysis, and what’d I find out?  Precipitation?  Doesn’t seem to matter.  Visibility?  Doesn’t seem to matter.  Wind?  It matters.  I can up my prediction of my oil usage by adding the total sustained wind speed to the equation.  In fact, it increases from an r-sqr of 10% to about 83%.  Yup, you read it right.  But the oil company doesn’t use wind speed data to calculate what I might use in the way of oil – just degree days.

What would it do to the company’s delivery planning if they didn’t have to haul around a full truck of oil because it hadn’t been very windy and we were all using less oil?  They could make fewer, or at least better timed visits to the customer.  What if they could get me to a 1/4 tank of oil before filling me back up instead of a 1/2 tank?  You heard me right, they fill me up when my tank gets to about half because who knows why.  I’m guessing because they can’t predict my oil use that well and if they targeted a refill at a 1/4 tank, I’d run out of oil sometimes.

I get about 7 to 8 visits from the oil company to fill me up each winter.  Each fill up is about half a tank of oil.  So, if they could fill me when I’m 3/4’s empty, they could save half a visit each time.  That means every 3 visits would be reduced to 2 visits.  Instead of 7 or 8 visits, I might have 5 or 6.  At $4.60+ a gallon for diesel to fill up the truck, saving 2 visits to every customer they have every year would be worth quite a bit.  And what’s horrible about it is a little extra free information is all they’d need to cut visits and still not have anyone run out of oil.

9/3/08 EDIT:  In the spirit of full disclosure, I did some more research on my oil usage because I paid for some upgrades to my system to improve efficiency.  I found that degree days makes a much better predictor than my prior experiment indicated.  This is especially true if I remove data about the warmer months – usually from late April/May when they fill me up at the end of the season through late Aug/early Sept when they fill me up for the start of the heating season. 

In fact, I just got a delivery today which was for an unusually large amount of oil probably because they can’t predict summer use.  I still see problems in the residuals suggesting that another variable is needed.  Regardless, my argument holds, if the oil company had a better model for predicting my use, they could fill me up less often and save lots of cash.  I should note, I have a very, very old house, so wind might have a bigger effect on my poorly insulated home than it does for most people.

Averages are not a bad thing

I was chatting with a coworker who had just been to some training class on process design.  In the class, the instructor told the class that averages were no good.  He said if we used averages, every person would be half man and half woman.  Actually, he said it in a much more crass way involving people’s genitals, but let’s leave it at that.

It’s among the stupidest things I’ve ever heard.  Averages are about the central tendency of a population, not what each thing in the population is.  For example, let’s say the average height of a person is 5′ 8″.  Just because the average is 5′ 8″ does not mean that every person in the population is that height.  Some will be shorter, some will be taller, and some will be 5′ 8″.

In the same way, if I randomly select 100 people out of a crowd (assuming that the world is 50/50 men and women and I’m not sure that’s true), I could expect to have 50 men and 50 women.  Yes, the population would be 50% male and 50% female, but that doesn’t mean that every individual shares that characteristic.

Yes, averages by themselves do not tell the whole story, but don’t let someone tell you that calculating an average is invalid just because they can imagine a silly situation due to a bimodal population.  In the long run, which is what central tendency is all about, the average will tell you where you’d end up.

Imagine you make bets worth $10.  You have a 50/50 chance of winning any given bet.  After 100 bets, you should have lost about 50 and won about 50 bets.  No single bet was both won and lost, yet the average still describes what happened to you.  Mind you, I’m ignoring normality and whether you should be using the median, but the point is, averages are about the larger population, not a single item in the population.

If your friend jumped off a bridge…

I can’t believe it’s come to this.  Process improvement decisions made solely on the basis of peer pressure!  I was talking with one of my coworkers who was on a committee (bad idea number one) to decide on a new process for something or other.  The actual thing doesn’t matter.

The “thing” is done by many teams already, albeit in somewhat different formats.  Since the committee is trying to come up with a standard, they needed to agree on what sections should be in the document and which shouldn’t.  Their basis for decision making, you ask?  If 3 groups are using the section, it’s now mandatory.  All the other sections of the document are gone.

No consideration to how the sections play together to form a complete package, no consideration if the section adds any value whatsoever.  Literally, “everyone’s doing it” so we should too.  How did it come to this?

I can imagine a scenario where everyone on the team really doesn’t want to be there, so being not particularly committed to the improvement and simply wanting the pain to end, they all struck a pact to use the popular vote as a means of determining what’s good or not.  This is not the way to choose people to make process improvements.  If you don’t have the interest or mental capacity to understand the potential impact of a process, don’t play with it.  While a company might survive a half-baked design or analysis document for a single project, what if the guidance you gave to everyone was shooting them in the foot?  The implications of standardizing on a bad process spawned out of laziness are horrific.

So there you have it, another exceptional example of how not to make a decision – advice you got from your mother.  <in my best motherly tone> “if your friend jumped off a bridge, would you do it?” </ motherly tone>

Change management comes to the small screen

So, maybe That Mitchell & Webb Look knew what they were doing, or not, but they’ve made a very funny short clip about change management.  Quite a lot of good information about change management is expressed in this little clip.

First off, we’ve got great examples of a stakeholder analysis.  There are 4 people involved: the Chieftain (I presume) who is a high influence but low impact individual and clearly supportive of Bronze.  We’ve got the two workers, the chipper and the stick-tie-er fellow who are low influence but high impact. The guy from the village down the road who wants to make the change isn’t part of the analysis, since he’d probably be doing such a thing if he was Six Sigma trained.

It’s obvious that the chieftain isn’t worried about the change, after all he stands to gain a great deal and doesn’t have to change at all.  The chipper, on the other hand, is going to lose his entire livelihood and retraining is a scary concept.  The stick-tie-er fellow is hesitant at first, but as soon as he learns that his job is safe then he’s fine with the change too.  Though I can imagine a follow on where the stick-tie-er would go through some resistance as tying a bronze axe head onto a stick is probably somewhat different than tying on a stone one.

It’s a good exercise in reading where someone might stand on a given situation.  Of course, there are probably lots of real life situations we all go through that parallel this ridiculous example, but I kind of like the silliness to separate the emotional response to the issue from the behaviors I want you to see.

Besides the stakeholder analysis, we can evaluate the individuals for the forms of resistance we see.  Hesitation on the part of the chipper, who asks a question about whether bronze needs to be chipped – fear of losing his responsibility.   And afterwards, when he’s talking about smelting being for the lads and being able to feed his family just fine with a stone tool – cynicism that bronze is really all that much better.

From there we can look at the (poor) change management skills that the guy from the next village over uses.  He attempts to reason with the chipper about how bronze is superior and that they should be using it just for that reason, but since change is a mostly emotional thing for people, this isn’t going to be effective.  A great Jonathan Swift quote on the subject: “you can’t reason someone out of something they weren’t reasoned into.”

Ok, ok, I know it is silly to evaluate a comedy clip this way, but it is one of the better examples of change management being done.

“I’m all done”

That’s what my daughter said to me at dinner tonight.  All in all, it wouldn’t be that interesting, but the context in which it was said sparked a thought for me. 

We were out to dinner at a little Italian restaurant down the street and I had ordered a pizza to share with her.  While she’s sitting there in her high chair (she’s two), she says to me (more or less) “sit with you, Daddy?”  Now, I’m more than happy to have my little one sit with me, but my wife and I use her requests as leverage to get what we want.

So, in response I said “when you’re all done with your pizza, you may sit with me.”  And she promptly put down her pizza and said “I’m all done.”  I sort of rolled my eyes and corrected my statement.

“Finish your pizza and you may sit with me,” I responded.

But that’s when it struck me.  Even my two year old has the understanding and natural behavior to literally meet the metric I set forth and do absolutely no more.  Since I had requested she be “all done” to sit with me, she simply declared herself all done.

That’s what we’re doing when we give our employees metrics that they can declare met – they simply do so because it’s in our nature to do no more.  It’s also why my second metric was better than the first.  I could observe that her pizza was eaten (or not) and did not require that she provide the distinction between all done and not all done.

Both of my requests meant the same thing.  I wanted her to eat her pizza, but the choice of how I phrased it made all the difference.  The same deal goes for measurements inside companies – the choice of how you measure something could make a huge difference in whether you get what you asked for or get what you want. 

By the way, my little one did eat the rest of her pizza and claimed her reward – plus an ice cream on the way home.  I’m always pleased when my little one teaches me something new about the world.