Tools amplify processes

Maybe this is old news to some people, and it probably should be old news to me.  I was listening in on a meeting today regarding the use of some tool we were going to use to store our requirements.  I say listening in, because the word “attending” would indicate that I was actively participating in this meeting.  I’m not sure why mind was adrift, but I think it was because I had decided that the presenter saw the tool as a silver bullet to our requirements problems.  Oh, and he said that we’d be having yet another tool brought in-house to report defects.  As if we didn’t already have three ways currently…

There are people out there who thinks tools solve process problems.  I think that’s partially true, but it depends on the process.  Tools don’t so much improve processes as they amplify processes.  What’s the difference, you ask?  If I have a good process and then I put it into a tool I get more “good” out of the process.  Tools can increase efficiency when doing something well.

On the flip side, tools amplify bad things just as well as good things.  If I take a bad process and put a tool on top of it, I simply create more “bad” stuff, more quickly and with a false sense of security because, “by golly, I’ve got a tool and tools solve problems!”

So let’s get back to why I was ignoring this presentation about the latest requirements tool (ReqPro, by the way, in case you were curious).  We already know that we have a faulty requirements process.  Requirements issues make up the bulk of defects that are reported.  If we are writing vague requirements today, a tool does nothing to create better requirements.  All it does is speed along the process of creating vague requirements.  Wouldn’t it be better if we created fewer but better requirements than gobs of sloppy, vague requirements?

I don’t want you to think that I don’t like tools.  I’ve heard other six sigma people say “technology does not solve problems” and I do not completely agree.  It is the damage that technology can cause in the wrong hands that makes that statement true.  Technology has an important place in software development processes.

Technology enforces and enables process.  If it is enforcing a good process, then tools like workflow management and source control (especially the two linked together) make sure the right things happen at the right time in the right order.  Technology increased the speed at which documents were getting reviewed at my old company.  Technology assured that neat little packages of documents and code were kept together.  Had I built a bad process, then technology would have assured we did a bad job consistently.  Technology would have increased the rate that bad documents zipped around the office.

Before technology could be employed the right process needed to be in place.  Technology doesn’t fix processes.  Technology amplifies processes, good, bad or otherwise.  So, if you’d like to see your bad process go awry in bold relief, get a tool.  You’d be amazed at how quickly you can make a big pile of junk with a bad process and some technology!

If you can’t distort the data, just don’t look at it

A sad reminder of the state of affairs comes from Curious Cat Management when it comes to conflicts of interest between the data and one’s own goals.  It is challenging to act with integrity when you are able to manipulate the facts to meet your own ends.  Just yesterday I ran into a baffling extension of distorting the data:  if you have data which tells a bad story about your own organization, hide it!

In my case, I was discussing a proposed MBF with a superior of mine.  I think like many organizations that deliver services we rely on three key areas of measurement:  Speed of delivery, cost of delivery and quality of delivery.  For some reason, we rarely measure the functionality of the delivery, but that’s another story…

Ignoring speed for the moment, there exists a paradox between the cost and quality aspects of delivery.  It is generally thought that driving up quality will result in an increase in cost as well.  The innovation in a process is breaking this supposed relationship – drive up quality and drive down cost.

For that reason, I proposed two measures as the basis for our MBF.  The first we already use today in some form:  defects per thousand lines of code (KLOC).  My proposal for the second was cost per KLOC.  I’m going to put aside the concerns about whether KLOC is a valid way to measure productivity.  While function point analysis resolves many of concerns regarding lines of code metrics, our organization lacks the maturity to think in those terms.  Fortunately, Capers Jones gives us a nice table in Applied Software Measurement which allows us to compare differing languages on a level plane.

Regardless, we don’t measure productivity at all today.  To me, it’s just obvious – any measurement of productivity is better than having no measurement at all.  Unless of course, you are heading up the organization whose productivity is abysmal.  And that is what I ran into the other day.  Why was my proposed MBF nixed?  Had I stumbled onto something that my superior did not want to share?

I’m still seething that someone would turn a blind eye to a productivity problem for fear of how it might reflect on them.  People are not stupid and even if you choose to never measure something then you lack any basis to dispel myths about your productivity or compare it to your peers.  Sure, it sucks to be the worst, but this is also your motivation to not be in last place.  The presence of data (even in the worst MBFs) drives action.  The lack of data allows you to put your head back in the sand and pretend it isn’t there.  It is there, but if it helps you sleep better at night to ignore the problem, go right ahead.  The problem will be waiting for you when you come back to work the next day as long as you refuse to act on it.

No measurement, no project

It seems to be a common theme in dealing with improvements in the Systems world that we never have any data.  There’s a bunch of reasons why not:

1. People don’t think to measure things.  I think this is fairly common until you have a problem and then usually people only measure the output measures of the process.
2. Systems produces fairly few widgets.  Unlike a true factory, I’ve worked in organizations that put out 30-40 new features a year or less.  With that in mind, I’d have to wait at least a year to know if a change I made is making any difference, maybe even two years – one year for the baseline data and another for the changed process data.

No business is going to wait 2 years to find out if a change worked.  And darn it all, this lack of data is getting in our way of progress!

So, when one of my GBs came to me recently and said “someone told me I won’t be able to get any data for my project, can I just estimate the baseline?” I responded with “if you don’t have measurements, you don’t have a project.”

If humans can detect the presence and figure out the size of planets orbiting stars hundreds or thousands of light years away based on how the star wobbles in response, surely we can detect the presence of a defect created within the group of cubicles we live in!  Even if we cannot observe the defect being created directly it must have some impact on the world that we can examine.

For example, let’s take production defects.  When something goes wrong in production, a very unhappy customer tells one of our phone reps about it.  In turn, the phone rep logs the defect into some defect tracking system with a bare minimum of information. Eventually, that defect makes it into the hands of a maintenance person who fixes the defect and closes the issue in the tracking system.  No where along the line is that production defect ever tied back to the project that caused it.

As a result, we are incapable of directly observing the impact of projects on production.  We know that projects cause defects in production, but since there are bunches of projects making changes to production every month it is impossible to tell which project caused what defect.

We can, however, tie pre-production defects back to the projects that caused them.  When our testers find a defect during testing, they log a defect which includes the project they were testing.  I’ve seen two models of defect detection for us:

In pattern A, for every defect we find, more simply get by us.  Bigger projects simply produce more bugs than smaller projects.  We find some number of bugs, but never all of them.  In pattern B, for every defect we find, less gets into production.  This is the pattern you’d expect from an effective QA organization. What’s great about either pattern is that there is a relationship between what we find in production and what we find in testing.

As a result of the inability to tie prod defects back to the project, we report a number called defect containment each month (like the chart above).  This is a fairly commonly reported output measure.  The problem is how we calculate it.  Containment is N / (N+P) where N is non-production defects opened in a given month and P is production defects opened in the same month.  The thing is, the code we are currently testing in June is not the code that is in production at the time.  Relating the two values together is meaningless.  If you do a correlation test between the non prod and prod defects in the same month you’ll find it has essentially no relationship.

But this is what happens when you can’t tie a defect back to the project that caused it.  So how do I know that we have two patterns of defect detection?  Indirect observation, that’s how! 

1. I know that the code is being tested in June. 
2. I know that 100 defects were reported against all projects being tested in June. 
3. I know that the code being tested in June will go live in August. 

Therefore, I can try to find a relationship between the June testing defects and August production defects.  When I do this, I find a very strong correlation between non-production defects and prod defects.  Sure, it’s not perfect.  Some defects may not be found until September, October or even later.  But the reality is that most of our production defects crop up relatively quickly.

Getting back to my Green Belt’s conundrum, you can see how I would say there was no project if there were no measurements.  Just because you can’t put your finger right on the data doesn’t mean there isn’t a way of seeing it indirectly.

Quit your day job

Though I have said in a prior post that “showing up” for work by just simply responding to requests (good, bad or indifferent) goes a long way to being a good employee in my estimation, it is certainly no way to succeed in an office.

I’m going to argue that, in fact, doing the job you are asked to do will most certainly garner you a satisfactory review but is no way to get ahead.  If you were to show up on the job working for me, my expectation is that you would do the work I assigned to you.  It would be done properly and on time.  That is what I expect you to do.  And yet, we find time and time again that people in exchange for doing as they are asked expect to have praise heaped upon them and to be promoted or get significant raises.  Why are they surprised when this doesn’t happen?

If you have a child, you are familiar with this concept.  If I asked my kid to clean his/her room then I would expect that it would be done promptly and that “cleaning” does not mean shove everything under the bed or into the closet.  That doesn’t count as cleaning.  What would impress me was if my kid cleaned his/her room without me asking for it to be done.  I have an expectation that everything I ask to have done would be done.  It is the things that I do not ask for that impress me.

Now, that doesn’t mean when I ask for a clean room that I expect a spotless room in which the lint has been collected from the corners.  I have reasonable sense of what a clean room looks like to me.

As an employer, I believe we all have an idea of what a reasonable job looks like.  Here’s my proposal: for the stuff you are asked to do, do a reasonable job and no more.  Then use the free time to do something impressive and unprompted.  No matter how meticulous you are doing your “day job” you can only make it so good.  You eventually reach the point of diminishing returns, where putting more effort into your work gets you less and less improvement.  Spend enough time on it and people will start to wonder why you can’t get your work done.

Don’t let work expand to fill the time allotted.  Do a decent job at it and then use the rest of the time to do something new and unimagined.  Provide your boss with information s/he didn’t know was out there.  Don’t spend time perfecting status reports.  In the end, it was asked for, so you have to do it, but nobody will care or remember anything more than the fact that you did your status reports on time.

The good idea smoke test

I was talking with my boss today about her husband’s business.  As an analytical person like me, her husband often turns to her to provide charts and graphs showing how his business has made a difference to the clients that hired him.  Of course there’s an issue of impartiality with the charts and graphs he has produced.  It is in his best interest to make the results look good or he’d be out of a job.

The conversation diverged to how my boss consistently questions the validity of his ideas.  He says talking to her is like “defending a thesis.”  For me, like my boss, it is fun for us just to pick apart ideas, turn them upside down, reassemble them and see if they still hold water.  It is the life of a cynic.  I have a hard time believing there is anything that is truly a good idea.

I have come to realize, however, that there’s a great smoke test for a good idea.  I will define “good” to mean an idea that is worth exploring more deeply.

I’ll give you an example going back to my post about Unintentionally Fixed Price Development.  I finally got the chance to present my findings to a larger group.  Because our costs are so predictable, I was able to say “we don’t need to estimate anymore, I’ll just tell you what the price is.”  The reaction was exactly what I was looking for – unhappy people! 

“How could this be?” 

“Maybe my data was no good.” 

“Maybe it was just the sample of projects I had.”

“Estimating is necessary!  We must estimate!”

And that’s when I knew I had a good idea.  People react badly to ideas that create cognitive dissonance for them.  For the people in this room, estimating was something difficult, artful and requiring a great deal of skill to do well.  I was telling them it was easy, scientific and required no skill at all.  Ideas that create these issues for people are ideas that have not been explored yet.  If the idea was familiar, people wouldn’t flip out.

Of course, given my propensity to rant about Agile software development, one might argue that my own statement proves that Agile people are on to something.  The strong reaction that it elicits from me suggests that is the case.  What’s important to note is that the reaction suggests the idea is worth exploring.  That doesn’t mean that it’s still a good idea once it has been more thoroughly explored.

In fact, this was the case with the reaction to estimating.  Though I can predict the cost of a project with a great deal of accuracy, one would have to accept that the current state of affairs could remain the same.  Since I had shown the existence of overblown estimates, it was necessary to consider what dis/advantages we would get by giving up estimating.  Since my formula did nothing to reduce the cost of projects or produce more aggressive estimates it failed to meet some basic needs for the company.

There is a difference between a good idea and a ridiculous one.  You can’t simply choose the opposite of whatever is happening today and call it a good idea.  You will get a bad reaction, but it won’t be the same.  I wish I knew how to explain the difference between being ignored/laughed out of the room for lack of credibility and getting a very similar reaction because you offend someone’s world view.  You’ll know it when you see it.

The project charter should be last

In the six sigma program where I work, I have 3 green belts (GBs) that I am mentoring through training.  It isn’t until you try and teach something to someone else that you really understand it.  And that’s why I had my little epiphany today.

In our SS organization, they train you to write the project charter first.  In fact, they insist on this so much that you have to submit a viable charter just to get into training.  Our charter contains the business opportunity, the estimated benefits, the goals and target measures and a timeline to complete the work.  Today, while prepping my GBs for their define tollgates I realized that we had been teaching this all backwards!

We have an issue of the chicken and the egg.  Officially, you have to have a project in order to do the define phase, but in order to determine if you have a project you need to get through the define phase.  How else will you know if you have a project?

What we’re teaching today is:  Charter and then (in no particular order, except where one naturally follows), VOC, CTQs, SIPOC, Sponsor Contract, Stakeholder Analysis, Risk Analysis, Financial Benefit Calculator and a handful of other tools.  This makes no sense to me!

Why do you decide to to a project?  It’s because you have already determined that there’s some problem that requires addressing.  But if you jump right into the charter then you’ve already made numerous conclusions about the project.  First and foremost, that there is a project at all.  How do you know you need to act?  What structure does the organization have in place to listen to its customers – surveys, listening posts, interviews, focus groups?  What are those structures telling you? 

First should be VOC.  An unhappy customer is the reason you are working on a project at all.  Second should be CTQs.  You can’t write a charter about improving unless you know what you are improving.  Then and only then should you be writing a project charter.  If you are writing project charters on a whim, what are you going to do about VOC? 

What is happening is that people go out and get VOC to fit their project charter.  We ask leading questions like “don’t you think there’s a problem with our system designs?” and “wouldn’t it be great if we had a standard design?”  Inevitably people say “yes, that’d be great!” because that’s what people do when faced with a leading question.  And then we say “hey, we have a problem here, let’s go forward with our project charter!”  Welcome the Texas Sharpshooter Fallacy.

One of the things that has always bothered me about the scientific methodology (despite me believing that it is a naturally occurring pattern of behavior) is that we begin with a hypothesis.  And being unwilling to fail at anything, all of us are inclined to make the data fit the conclusion we have already come to.  It takes a strong person to admit that s/he found nothing.  To be able to really and truly make positive change, we must be willing to listen to all the VOC and figure out what people are really telling us, not fitting what they are telling us into a predefined package.  It is our customers who define success.  By writing the charter first we have already decided ourselves what success looks like.

If we cannot have an open mind approaching the problem with the charter coming first, then we need to change the approach.  We’re setting ourselves up for failure to begin with.

Avoiding Non-compliance

If you’ve built yourself a good process and proved that it works, you want people to follow the process.  You know that it makes a difference and it is really frustrating when people continue to cause problems by not doing “what’s best for them.”  Like a couple of prior posts, this one is about why people don’t do what you want them to do.

 I’ve already been down the People Are Selfish road so I’m not going to rehash it.  Let’s take that for granted.  Today I wanted to give you another option for getting people to do what you know to be the right thing.

Here’s the idea, and it’s super-simple.  Make it easier to do the right thing than the wrong thing.  “What!?!?” you say, “I’m not paying for that kind of advice!”  Of course, you don’t pay me anything, so…

When process compliance is a problem, you have to look at what it takes to get the process right.  Going back to that people are selfish (and lazy) thing, if there’s a shortcut to be taken then people will take it.  If taking that shortcut hurts the performance of the process, then you must make it hurt more to go down that path than to not. 

Take, for example, my old team at my last job.  No software developer likes writing documentation (or thinking through a problem) and they want to get right down to coding.  I already knew that this was going to be an issue.  The process I had designed worked because it relied on peer review to assure that everything was getting thought through.  That also meant that being sloppy was not an option for my employees.

Why were people sloppy?  Well, they figured if they didn’t get the design right in the first place that they’d just go back and fix the code later and that fixing the code later would be easier than figuring out the right thing now.  Of course, that’s not true.  Finding a bug later in the process results in effort that is orders of magnitude worse than finding and fixing it early on.

So, what could I do to make people do the right thing?  I couldn’t make people write good documentation.  However, because I had set up controlled access to the source control I could make them write more documentation.  In fact, this is exactly what I did. 

If a person wrote a defect and it made it to QA, they would have to write a complete set of documentation in order to fix the bug.  That meant even for the most simple one-line code defect I required an analysis document, an analysis review by a peer, a design document, a design document review by a peer, the code, a code review document by a peer, and a test results document.  It is really annoying to fill out all those documents for a single line code change.  It is so annoying, in fact, that it drives down the defect rate.  The cost to the developer for screwing up is now significantly greater than the cost to get it right in the first place.

Developers, or anyone for that matter, do not see the costs they incur on someone else for their screw-up.  If my laziness results in the need for a full-time support person, it doesn’t hurt me directly at all.  But, if my laziness creates work for me, then I’m less inclined to be lazy.  I hate doing stuff over.  I hate filling out documentation.  I hate being punished for something that I had the capability to avoid.

So, while what I really wanted was for people to do a good job at the documentation there was no way I could enforce it.  But, since doing a bad job at it could result in a significant punishment of even more documentation, I had an indirect means of getting what I wanted in the first place. 

And that is making it easier to do the right thing rather than the wrong thing.

“Management by Fact” not “Reaction by Fact”

NOTE:  I reworked this post based upon more recent similar experiences.  Same concept, different story to tell it. 

I ran into a disturbing activity this afternoon at work.  We were discussing an MBF for a testing processes and looking at the trailing indicator – incidents in production.  The number of incidents in production rose last month “unexpectedly.”   By unexpectedly, I mean the process owner and customer did not like the number.  Naturally, I wanted to go look at our leading indicators to see what changed.

Our process owner did not.  He wanted to get the data for each and every ticket and look at it and figure out a root cause and then make process changes to prevent each of those kinds of defects from happening again.  Ack!  Where did we go wrong explaining this MBF?

The point of having an MBF is understanding the critical few levers that you have to change the process performance.  If you see a leading indicator going in the wrong direction, you can rest assured (if you’ve done your job right) that the output result will go south as well.  But did our process owner look at the leading indicators or the countermeasures plan to see what to do?  No, he wanted to look at the root cause each ticket and see how to prevent that specific issue from occurring again.

Looking at each ticket is no way to go about managing your process!  Why not?  This is reacting, not managing.  Leading indicators warn you of something to come and if you aren’t seeing it until after it happened, you aren’t looking. 

Let’s say I made candy for a living.  Each month I might measure the number of defective packages of candy.  So, my output measure would be “defective packages of candy per month.”  Then, were it a real MBF, I’d have leading indicators that would be affecting my good candy package yield.  Let’s say, sugar quality, time in the flavoring machine and number of rats scurrying around the factory floor.  If one of these leading indicators were to go the wrong direction, say an increase in the number of rats in the factory, I could conclude that I was likely to have a lot of defective (in this case, probably half eaten) packages of candy.

But rather than do that, we see MBFs where the number of defective packages of candy is the trailing indicator (this is still good) and that our “leading indicator” is defective packages of candy by root cause.  It’s the same data as the first chart, except stratified by candy packages that are “too sour”, “flavorless”, or “half-eaten” and maybe some combination of the three!

Sure, I now know what went wrong with each package of candy, but I haven’t done anything to prevent it.  I could have taken action had I known that an increase in rats was going to cause more candy packages to get eaten.  And I could have done it BEFORE the candy got eaten.  Now all I can do is look at my root cause data and say “oh, maybe we should put out some rat traps” or maybe since we failed to keep the rat population in check in the first place there is nothing left to do but close up shop.  The rats are running the factory now.

The next time you go to make an MBF, don’t provide endless stratifications of the trailing indicator just because some uninformed user thinks that’s how you manage a process.  Educate them.  It’s called “Management by Fact” not “Reaction by Fact” for a reason.

Build it or buy it?

Oftentimes this is a question when it comes to software itself.  People are always trying to determine whether they should build a given application or buy something off the shelf and just use it.  A great blog entry speaks to the exact issue.  There is a school of thought that software should fit the process and not the other way around.

I’m questioning if there’s another level this can be extended to.  Do you “buy” a software development process (SDP) or build your own?  There’s a broad range of SDPs out there today so it certainly would be easy to take one and use it, right?  Why would you ever start from scratch and roll your own?

For one, I think that most large processes are freakish monoliths that are more marvels (and I mean “marvel” in the least generous of terms) of engineering than practical applications.  CMM, for example, though not an SDP per se, is a great example.  I have been on a team who was attempting to reach CMM level 3.  We abandoned the effort because it killed our flexibility.  Now, it may have been a lack of creativity on our part, or it just may be that the process was more demanding than we needed to be effective.  That team of folks had a very lean development process and insanely low defect rates.  The system of about 100,000 lines of code had 12 known low-criticality defects.  It was a medical device, so having defects was a big issue.  And yet, we didn’t benefit from having a bulkier process for getting our work done.  Defect containment rates were consistently 100% and defects found during test were usually very minimal.  The existing process was very, very sound.

And yet, someone thought, “why are we using our own process?  There’s one out there we could use.”  Can and should are two different things.  So here I am at my new company, almost 12 years later and I’m hearing “we should switch to Agile.”  Putting aside my rants re: Agile in general, what is driving us to even think about it? 

Do we have problems with our development?  Sure, who doesn’t. 

Do we have defect rates we don’t like?  Sure. 

Do we have schedules we don’t like?  Sure. 

But we have a process which the company has been using for years.  How is it we are unwilling to understand what we have but willing to buy what someone else is selling?  If we don’t understand what drives the performance of the current process, what makes us think that going “Agile” is going to solve any of our problems.  What if our current process is good, but the key driver of performance is the below-average employees we hired?  An Agile implementation would suffer greatly in that reality, based on recommendations from Barry Boehm’s work.

A favorite refrain in the Six Sigma world (at least the one I’m in) is to “not boil the ocean” which has led us to attempt to chip away at the efficiency and quality issues in the organization.  Yet the process which our customer sees is from request to installed functionality.  Why are we fixing just software design or coding or unit testing?  If we understood the real end-to-end process we could optimize across the hand-offs and truly make a difference.

There are no silver bullets.  Are we “buying” a process because we’ve given up and we don’t understand the one we have? 

Are you?