I hope the new site content will be even more focused on the topic of LEAN in software, and I hope you’ll join me as a reader at my new home.  Thanks!

It may still be the right thing to do in the absence of our ability to change anything else, but I now believe that it may create more problems than it solves in the long term. What if we were to improve our test organization’s capabilities until they were able to catch, say 99% of all bugs that made it to them. They’d have an impressive defect containment rate, that’s for sure. But they’d also have another problem. To the end user, our customer, they don’t care how we achieved such high levels of quality. They only care that the product is of quality. And what’s more, what incentive does development have to be better if they know that testing will catch their bugs. The goal is being met, albeit in a wasteful way.

It’s not that the quality isn’t good enough. It’s not that you might be able to achieve it for a reasonable price and time. It’s that you could be doing it better. It’s better to not create the defect at all. You may be capable of doing great things within the boundaries of the process you control, but an even better solution which nobody will go looking for may exist. It’s a solution nobody will ever find if you continue to cover for the up-stream process failures.

I’m not sure it’s the absolute right answer, but I now have a philosophy. Percents should never be used to talk about rate of change. For example, if someone says “we had a 100% increase in defects from last month” you don’t know if that means you went from one thousand defects to two thousand or that you went from one defect to two. Both are 100% increases. We often see this in sales literature – “realize a 50% gain in efficiency!” But how much effort were you putting into it in the first place? 50% might be a lot, or it might be a large proportion of nothing.

I like percents for comparing groups of things. What percent of defects were requirements vs. code vs. design? Think pareto charts. I also like breaking things down by percent when looking at groups over time because each month might have a different number of defects, sometimes quite significantly, and using the percentages helps expose whether the groups are stable over time or not.

But not so much for change month to month. I think that just ends up being misleading.

Now, I’ll admit that Bill’s Pizza (note the sad missing apostrophe) was probably not attempting to demand shape with this promotion, but in effect they have. By offering a lower price earlier, they are essentially encouraging early orders, and presumably less demand around 9pm when they want to clean up and close. The delta between the best price and worst price is significant – an 80% premium to order at the last minute versus the first.

This kind of variable pricing might be a great way to smooth out the lunch rush – make it more expensive from 11:30 – 1pm, but somewhat cheaper just before and after. And a smaller rush means less people needed to handle peak hours. A creative way to demand shape that gives the customer something for changing their behavior. This is quite unlike the earlier experience I had with Intuit that I wrote about where the “don’t call us, we’ll call you” mechanism didn’t so much shape demand as it forced me to sit around my house. In Bill’s Pizza’s case, you can still get a pizza anytime you want, but at least you are encouraged to order earlier.

Finally, as we got to the end of this meeting, the facilitator asked to the audience “what does being a leader mean?” It was sort of a wrap up question. People said things like “listens to people”, “stands up for what they know is right”, “models the behaviors,” and so on. And after a bit of this nonsense, the faciliatator asks “what do you all think of these things?”

I raised my hand. I said (more or less) “these are platitudes. We all know these are good things, but I don’t and I’m guessing no one else, knows how to do these things.” Essentially, we get what ought to be done, but it’s like emotional intelligence. Even if you know what the right answers are, you don’t necessarily act that way in a real life situation.

This is why leadership has so many books written about it. This is why leaders are so revered. If we all knew how to do this, if we could all attend a seminar and really become leaders then leadership wouldn’t be anything special. The reality is, we don’t know how to train for leadership. We barely know how to select for it. We seem to know leadership when we see it, but we can’t articulate what about a person makes them one. It’s a certain je ne sais quoi.

But it also poses a problem. If we can’t adequately train it and can hardly select for it, what should we do about it? Do we just accept our fate? Perhaps not. Perhaps instead of trying to make everyone into leaders we shouldn’t try at all. Some will naturally emerge and that’s probably good enough. For everyone else, what’s so wrong with being a high quality follower? What’s so bad about just being good at doing your job well? Why does everyone need to lead anyway? If everyone did, who is going to follow? You can’t lead if there’s nobody to lead anyway.

The conversation around whatever the potential issue is continues for a while. It finally ends with “send me the data so I can take a look at it.” “Sure, I’ll get it to you tomorrow,“ is the reply.

Tomorrow comes and goes. The week comes and goes. So does the month. No data. The barrier for entry to getting assistance is so low, just send an email and attach the data (excel workbook, CSV file, SQL query to get at the data, etc.) and you could get some input. Why does the email never come?

This is what someone who is not serious about change looks like. Tachii Ohno and his disciples set a much higher bar as test of seriousness. When they’d show up in a factory, once you had been persistent enough to actually get them to show up, they’d insist you change some things right then and there. If you balked, presumably they were gone.

When someone approaches you for help, and you do all the work for them (like getting their data or holding their root cause meetings for them, or whatever), there is no hurdle for entry, no minimal test of seriousness. As the expert, you are hand holding them through even getting started. Where’s the desperation to change if you do that? Where’s the seriousness that indicates they really want help?

Declining to help may be a critical first step to really helping someone. Until they reach the point that they want change so badly that they’ll come back after being rebuffed, you know you have someone serious about changing. You must set some barrier to entry. If you will gleefully charge into any battle of attrition on anyone’s behalf, the only person who is going to get worn down and killed is you.

Fortunately, he’s a good kid (and conscientious, if not a bit quick to the keyboard) about his statements so he revised it to something a bit more readable. He also promptly complained “I feel like I’m rewriting an essay.”

My response? “But the result is better when you can read it. While science can tell us many things about the world, it is regrettably useless without a language with which to communicate the knowledge gained to others.”

And it was that response that led me to write this blog entry for you. We probably underrate the importance of being able to clearly communicate an idea to another person. The problems are endless:

1. We have horrid sentence structure, spelling, etc. All basics that we can learn to overcome.
2. We mumble, stutter or otherwise stumble over our words when we speak (if the information is being presented orally). The ratio of signal to noise drops and people lose interest. In writing, we ramble.
3. We assume knowledge that the listener does not have. Confusion arises and we lose the otherwise interested party.
4. We dumb it down too far and the listener has far more knowledge than we assume they have.

I could keep going on and on, I’m sure, but you get my gist. No matter how good your research on the problem at hand is, no matter how much new information you have to share, nobody is going to hear it if you can’t communicate it adequately. Practice communicating your ideas. Constantly.

Indeed, in theory LOC works just fine to measure the size of software, but they work terribly in practice. Why? Because as soon as you tell a developer that their productivity will be measured in lines of code written, they simply write more (and less efficient) lines of code. Suddenly being thoughtful about how to write code is not nearly as important as producing a lot of it.

FP, on the other hand, do not seem to have this issue. A developer can’t simply create more FP on demand. The FP count is bound by the size of the request; can be counted from the requirements and thus is freed from the influence of the developer. So, why, in studies then, are FP and LOC so highly correlated? Why would this happen?

As I was driving home (a common time for me to think, I’m sure you’ve noticed), a possible answer came to me. FP and LOC do measure the same thing in a theoretical or uninfluenced world. That is, if you go back and collect data about a bunch of projects you’ll find that FP and LOC are highly correlated. Because none of the historical projects were incented to produce unnecessary lines of code, the relationship holds up nicely when collecting baseline data on both elements.

The problem isn’t the baseline. It’s the next experiment we conduct that’s an issue. We go ahead with a FP counting system for our new projects. And then, we come back and count LOC as well and find them still to be highly correlated. And for some reason we’re upset! Why’d we spend all the time and money on a FP counting system if LOC would work just fine!?!? What we should be doing is running two totally different pilots. One where we measure and incent upon FP and one where we measure and incent upon LOC. Then, after a while, go back to population 1 and collect the LOC data where we incented behaviors on FP and go back to population 2 and collect the FP data where we incented behavior on LOC.

I bet you’d find that the correlation in population 2 would be broken while the correlation in population 1 would hold. Why? Because once developers are incented to do the wrong thing, LOC is no longer a meaningful measure. Great in theory, terrible in practice. In this space it is possible for the code to lose relationship to the size of the problem because we encourage the wrong behavior. LOC don’t have to be related to FP, but if we don’t muck with the system, they will be.

On the other hand FP are independent of the developer’s control. Therefore, they find no reason in such a measurement environment to artificially increase the lines of code. Writing extra code gains them no benefit and so they don’t do it.

The important issue here isn’t that LOC can’t measure the same thing FP can. It can and likely does. The important issue is that FP eliminates the incentive system that developers get when you measure on LOC. The ability to independently assess how much is being built is the important part. The critical issue is the difference between what works well in theory and what works well in practice.

Now, as we log-transform the data, we begin to see what we’re really looking at. First, on one axis, as below. Note, you can now see the data points but they seem to be all against the left edge and they look like they curve. When you log transform a single axis, linear data will take on a curved shape.

When you then log transform it on both axes, the curved shape is removed and we can see the data as it really is, linear, but much better exposed. Indeed, the data I created was based on a simple linear formula: Y = X * 100.

So, rule of thumb number 1 is, log transform both axes if you are going to do it at all. This of course becomes a problem when there are 0 values in your data, since you can’t log transform 0, but I’m not getting into that.

The second thing to be aware of is that log transformed data, if the original data has uniform variability at all levels of X will not appear that way once transformed. Instead, as the values get larger, the data points should appear to converge. Take for example, this simple example. Below we have two charts of data. On the first chart, the two data sets are:

Y = x * 10 and Y = x * 200. If you were to draw these on untransformed axes, the lines formed by the data would clearly diverge. Where X = 1, Y would be 10 and 200 respectively (a gap of 190). Where X = 100, Y would be 1000 and 20000 (a gap of 19,000). However, once you log transform the data, the lines appear parallel like the first chart below. Visually, you might say that if these lines represented the boundaries of some real scatter plot that the data appears to be homoscedastic (having uniform variability), but it isn’t! It’s a visual trick. Don’t get fooled by it!

By comparison, the second chart shows two lines which would be parallel on a chart with untransformed axes. One formula is Y = x + 10 while the other is Y = x + 500. At any level of X, the gap between the two data points is the same, 490 units. When you log transform this data on both axes you can see how it appears to converge. You might think that the data that exhibited this shape is heteroscedastic, but it isn’t! Again, you’d be fooled by the visual appearance of log-transformed data.

Why does this matter? Once you log transform data you often forget what you are seeing. You may actually have a spray pattern and think it is a simple linear correlation. This is often the case in software data, where larger projects may be generally more expensive/defect laden/time consuming than smaller projects, but the variability of the result is much, much higher. We get fooled into believing that we have a uniform linear relationship between any dependent variable we have and the outcome we are looking for when we do not. Don’t let making charts readable make you forget that log transformations, while exposing the data, change the shape of it in important ways anyway.

Anyway, there was a CEB blog entry (yes, they blog) regarding a question one of the individuals often receives about organizational structure. The question is essentially “is it better to have project managers report to the PMO via a dotted line relationship or straight line relationship?” The PMO, or Project Management Office, is typically an organizational body responsible for the end result of all projects plus overall portfolio reporting etc. One might conclude that having a PMO necessarily means that all project managers ought to report to that organization, but it often isn’t the case. Instead, project managers report up through some department and may have some obligation to the PMO for reporting status, project outcome, etc.

The author of the blog entry essentially said “who cares where project managers report?” The issue that we’ve found through study isn’t whether your managers report to the PMO directly or indirectly. The critical factor is did you get the right project managers in the job? Do they have the right skills? The right personality type? The drive to do good for the company?

We obsess over stupid stuff. Stuff that doesn’t matter in any significant amount compared to the stuff that although hard to figure out would actually make the difference. If you think you’re going to see step change in your organization by changing where the project managers report to, you’re fooling yourself, is essentially what the CEB is saying. Instead, change that matters is probably going to be either difficult to do or at least difficult to find. If massive improvement were staring us in the face and simply required the flip of a line on an org chart, everyone would have done it already. Unfortunately, you’re going to have to go hunting for the rest of the secrets.