The consumer is always right

I was sitting through a meeting the other day and we were talking about the new test case management system that we were installing. It’s nothing special, but one of the features it offers is the ability to track defects along with the test cases. The thing is, as a company we already have 3 other defect tracking systems from different vendors, none of which are integrated with our test case management system. True, it is an annoyance and certainly adds some cost to quality assurance to record the defect when you have to go into a different system.

However, the issue I have is that quality assurance designed the workflow and fields that would go into the defect tracking system. Quality assurance is a (sort-of) user of the tool. They have to enter data into it, so making it convenient for them to enter data is a good thing. However, they aren’t the primary consumer of the data – development is. The reason we enter bugs into the defect tracking system is (besides metrics) so that development gets a detailed account of what went wrong so they can fix it. Although we supply defects into the process, it is development that has to consume our input and use it.

So why then would you have QA define the input for the team who needs the data? It is illogical that QA should determine what information and in what format they’ll be giving it to development. Development, who needs adequate input to fix the bug, should be defining what inputs they need. QA could input more data than development needs if they felt they needed some data recorded too, provided it isn’t interfering with what development needs to get their job done.

Now, maybe development requests something that is ridiculous and QA ought to negotiate for a more acceptable request, but generally I believe that it is the consumer who gets to define what they need from their supplier. In our case, we didn’t ask development at all what they needed, we just built something. We didn’t ask development if they’d be willing to use our new defect tracking system.

I think, if it hasn’t been set out as a universal truth somewhere else before, that it ought to be done so now – the consumer defines the required input, not the supplier.

Uneventful change

As I was sitting here doing nothing, or effectively nothing (I was playing mini-games on my computer), for some reason a bunch of failed change events came into my head. As my prior post would indicate, I fail to make change a lot of the time, though not for a lack of trying.

To be fair, if I make a change and it isn’t better, then I hardly count it as making a change. Sure, something is different but not in a way that is meaningful to our customer.

Anyway, I digress. I was thinking about change events and why they always seem to be such a big event. It brought to mind a story about trying to build a new requirements process. Years ago, I worked for a team who was doing a process redesign for their requirements process. It was going to be this big multi-day event where we brought everyone offsite to hash out a new way to do requirements.

Never mind the fact that I believe this type of work really needs to be done scientifically as opposed to just throwing people into a room and saying “design something.” Anyway, I was tasked to go off and get a resource from this one team to attend. I approached the vice president of the team and said something along the lines of “I’d like to have a representative from your team attend this 3 day offsite to redesign the requirements process.”

“No,” he replied. I sat there and stared at him. I really didn’t know what to say. It wasn’t an angry stare on my part, it was more like a mouth agape bewildered stare. Kind of like the way my dog cocks his head to the side when something makes a funny noise. I’m pretty sure that’s what I looked like. Fortunately, he went on.

“We already redesigned this process once with Bob [a
coworker of mine]. Why should I send resources to do it again?”

Fair enough. It was true; Bob had tried to do this process redesign before. It had failed not because they were unable to come to an agreement over the change but because the team who was supposed to adopt the change refused. Foolishly, Bob had neglected a major stakeholder in the process. But, darn it all, we weren’t going to make that mistake again.

And yet we did. The mistake wasn’t trying to get all the people in the room, or doing it via an offsite, or doing it without understanding the existing process fully. The mistake was making change an event. An event is easy to resist. And event has a clear beginning and thus a clear line in the sand you can take a stand against.

What if change just snuck up on you? It happens all the time in our regular lives and we handle them in stride pretty well. Tools like twitter, Facebook, even the internet and email just creep into our lives without making a big scene. It’s hard to resist something that shows up quietly, that doesn’t really come knocking or announcing itself. It just shows up because it is better and more useful. Now that we have these things, we can’t imagine life without them.

Why can’t we approach process change this way? What if instead of a big-bang event to redesign the requirements process we just made some small quiet change to the way we did things? What if we just started inviting teams to JAR sessions. No training, no announcements that this was the new way to do things. Just invite them on some projects and see how they feel about it. If they like it, maybe we do it on some more projects. Eventually we’re doing it all the time and it becomes an accepted way of doing things.

There’s no real need, except potentially for speed purposes, to announce a process change. And given the enormous resistance that can be placed against an event, perhaps it is just better to sneak a little change for the better here or there. Do it often and eventually the process will be changed for the better, and significantly so.

Perhaps a better title for this posting would be “eventless change,” but the idea is the same. Change doesn’t have to be eventful for it to be impactful.

Not one of the critical few

Ingrained in the Six Sigma school of thought is the critical few – the 80/20 rule. It is an important rule. In practice, there are a handful of things which often allow you to make big leaps from an incapable process to a capable one. There are more subtle characteristics of the process which can be refined to continually improve the performance, but this isn’t step change, it is refinement. And then there’s a class of things that just don’t matter.

So as I sat today through a long, long meeting trying to define a process, I spent a lot of time thinking about the things that don’t matter. That may have been because that’s all anyone spent their time talking about. And as facilitators, we were enablers of this dragging on. Having been instructed to drive to a single standard process and toolset, we discussed every little one-off thing that people wanted to allow for in the process to see if we could squeeze them out. A day’s worth of 25 people’s time to design a process spent talking about the equivalent of the carpet color.

We wanted perfect compliance to the standard, and that meant a standard which was not necessarily all-inclusive (because some of these one-off requests were truly ridiculous by any standard). This is where I believe we got off track with process work. Process design is about controlling the critical few things which will make the difference in process performance.

But that is not what we were discussing. We were discussing nuances, oddball cases, odd uses of the process, and data elements that some teams wanted and others didn’t. We talked about the 1% and largely ignored the 99%. We talked about things that weren’t going to make the difference, whether they were one way or another.

To begin with, we didn’t know what was going to make the difference. We hadn’t studied the existing processes to understand what made them work – what really mattered and what didn’t. This created unnecessary room for debate because we were unable to bring adequate materials to the table to help the team work through their differences. We had little to no information on what mattered and what didn’t.

Instead of define-measure-analyze-improve-control we just went right into improve. And there we got bogged down discussing every little quirk, because we didn’t know what else we ought to be talking about. Or more importantly, what we shouldn’t be talking about.

Instead of a conversation that was “do we really need that? How many of our teams use that process step?” we could have instead said “sure, it doesn’t matter to me if you allow for that.” And we’d be saying that not because we didn’t care but because we actually knew what did matter. Everything else, the little things that we debated with the teams could have instead been bargaining chips that we could dole out in heaps and have given up basically nothing that really mattered. We could have had a strong position, not because we won all the arguments but because we knew which battles were worth fighting and which were worth conceding.

Had we known what things were not one of the critical few things, we could have appeared very agreeable and allowed the teams as much “leeway” in the process as they claimed they needed. All along we’d be giving up nothing. Nothing that really mattered anyway.

It’s a reminder why a thorough measurement and analysis of a process is important. It isn’t just discovering what the current state is (measurement), but it also understanding why it works (analysis). And from there, narrowing down the bits of process that really do matter, and just letting the rest go. Some things just don’t matter.

It’s a net, people

Ah nets, the greatest safety device possibly ever invented. And also very handy for catching fish. Or is it? In software development, we always talk about nets. Testing is a net to catch all the bugs that developers write. Peer review is a net to catch all the design flaws that analysts create (and possibly a net to catch code bugs as well). But a net it truly is, and nets, by design, have holes in them.

Nets catch big things, like big fish, and let the little ones through. That’s fine if you don’t mind the little fish escaping, but what if instead of being fish the little things getting by are defects? Sure, the really enormous fish, er defect, gets caught, but the little ones, the hundreds of thousands of little ones slip right through.

Nets serve a purpose, but they are imperfect and will always allow some things that are small enough to fit between the spaces to slip through. There are finer nets than others. General purpose nets, like peer review, tend to catch more than special purpose nets like regression testing. After all, peer review (in theory) looks for a broad range of things while regression testing only looks for what it already knows about.

But the purpose of my post is not to talk about the fineness of any net but instead to remind you that no matter how fine your net is, it is still a NET. And you really don’t want nets when it comes to software development, you want walls – barriers that allow NOTHING undesirable by. The barrier that we most often overlook in software development is error proofing the process.

Sure, for new code that’s hard to do, but many systems are configuration driven, and require changes to be made to many parts of the system to enable new functionality. We often need entries in the tables to create the foreign keys, entries in tables to join values together. Multiple rows of configuration working in concert make the right behavior. But what do we do? We write instructions on how to configure the system. Rather than do that, if there’s a limited number of possible results you want to support (say a bunch of client features), then either write the system with a single item of configuration (so you don’t have to worry about updating the right thing in every place) to make it work OR write a tool that makes sure you can configure all those settings with a single click. Error proof. Make it easy to do the right thing.

Walls, not nets, is where the answer lies.

Involve early or Just In Time?

Here’s something that keeps coming up at work that I’m not sure how to deal with. The Agile philosophy has the team entirely together all the time, including the business, developers, testers, etc. As a quality assurance organization, we constantly push to be involved earlier in the process (so much so that I think one day we’ll want to have an opinion on whether the business should invest in a given project or not). We all push to be involved early and often.

However, as far as I can tell, this is somewhat counter to LEAN production, which strives for effort as close to the point of demand (just in time manufacturing) as possible. Now, maybe I’m making a bad overextension, it’s entirely possible, but if we think of quality assurance as manufacturing test strategies, test plans, test cases, etc. then having them involved really early is the creation of inventory.

The earlier they are involved in the project the more likely they are to create their work product which is inadequate due to changes in the requirements, design, etc. later on. It is, in theory, wasteful to create anything earlier than absolutely necessary.

So, for my few readers bold enough to comment… what’s the best answer? Are the benefits of early involvement justified or do dedicated quality assurance activities (like testing) just need to be prepare for a Just In Time approach to manufacturing and executing testing?

Don’t unecessarily break cultural norms

Today I was in an all-day meeting which was the wrap-up for an Agile iteration. It’s a, what’s it called, reflection session (silly hippie name), aka post-mortem for the rest of us. Anyway, one part of that session was to perform plus-deltas. Plus-deltas are essentially a coarse way of getting feedback from the team about what they liked, what they didn’t like, what they want more of, etc.

For Agile projects, where the team decides the process, it’s part of being involved. I don’t have any issue with the doers being part of the solution. Quite to the opposite, LEAN philosophies insist upon it because it is the workers who know what isn’t working. But that’s where the similarity ends. In LEAN, once a best method is discovered, everyone does it until a better way is found. It becomes a form of local sub-optimization with continuous improvement being the goal for making changes. Agile, so far as I can tell from my reading, is more arbitrary. It’s whatever the team likes as opposed to whatever is currently known to work best. Maybe those two things are the same thing, but maybe not.

Anyway, that was a huge digression. So, we’re doing plus-deltas by writing up sticky notes and posting them on whiteboards inside key themes. Themes might be “the process”, “the tools we have”, “the people on the team”, and so on. As a little twist, the facilitator chooses 3 color sticky-notes and gives each sticky color a role. There are plusses (things we want to continue to do), deltas (things we want to stop doing or do differently) and he adds a third thing (which he calls “wannas”) which are things that we think we should start doing.

So if you were going to choose a color for each type of sticky-note based on what type of thing was going to be written on the sticky-note, what would you choose?

I’ll wait while you think about it…

[ insert Jeopardy music here ]

… ok. Let me guess, you chose the following:

Good things (plusses) go on green sticky-notes.

Bad things (deltas) go on red sticky-notes.

“Wanna” things go on some neutral color (maybe blue) or yellow.

Was I close? Did you ever consider putting good things on the red sticky-notes or bad things on the green sticky notes? No? Why not?

Because we have cultural norms about these colors. Green in the US means good while red means bad. That’s not true in China and Japan, where red is a good color. I’m not sure how they feel about green, but red isn’t a bad thing. Alas, for some reason this is what the facilitator chose:

Plusses are to be put on yellow sticky-notes.

Deltas are to be put on green sticky-notes.

“Wannas” are to be put on red sticky-notes.

Watching the teams work, you can imagine the confusion. The facilitator had to put up a cheat sheet about what color meant what. When the overhead projector went to sleep and our cheat sheet wasn’t visible, people were lost as to which color to use. Work stopped while people tried to remember which sticky color meant what. It was ridiculous, and completely avoidable.

In designing a process, there are things that you want to change, like the way teams work or even the corporate culture about reporting errors. And then there are things you should never change, like things that are cultural norms that would be extremely hard for people to not do the way they’ve always done. Regardless of how you feel about it, it probably isn’t sound advice to start making everyone walk on the left side of the hallway as a process improvement because it’s just going to screw people up unnecessarily. We have a norm about that that extends beyond the company, so don’t mess with it.

We’ve decided to do it the wasteful way

There are seven (or eight depending) forms of waste that LEAN recognizes:

1. Inventory
2. Overproduction
3. Over processing
4. Motion
5. Transportation
6. Defects
7. Waiting
8. (optional) Intellectual

Today, I learned from a friend of mine that we were going to deliberately introduce #6 as a valid way to behave.  Test Driven Development (TDD) is a development philosophy where you write a test case to fail, then write code until it works, and then execute the test case again to show that it now works.

Practically speaking, I get that we will probably never be able to eliminate all appraisal activities.  After all, unlike a faulty bolt which might cost us a nickle to replace, a faulty line of code could in theory cost us millions.  We appraise because it’s the only way we know how to check for correctness.  However, that doesn’t mean we have to accept appraisal as being necessary.  Far from it!

We know from the great research of Capers Jones and Barry Boehm that appraisal activities, especially testing are only about 40% effective per test type.  By comparison, visual inspection (aka peer review) can be 60-80% effective.  So why would you build a process around an ineffective practice?  And why, dear lord, why, would you run a test case you KNOW is going to fail.  You wrote it because it would fail.  If you’re writing test cases you don’t know will fail, are you paying any attention to what your system can and cannot do?!?!?

Now, one might argue that TDD is simply a manifestation of really solid use cases.  If we wrote adequately detailed use cases, they could in theory serve as the tests to appraise the system.  Fair enough, but that assumes that we are willing to exchange some level of intrinsic knowledge in our developers for complete and perfect specifications from our business.  The business will now tell us exactly how it has to work.  We can also reasonably assume that for TDD to be effective, you can’t just write the positive test case and show that it works.  You must also write all the negative cases, the exception cases where the system should deal with user error.  After all, those things must be specified to fail as well so that you can write code so they won’t fail.

Alas, this methodology is faulty to begin with.  There are known better ways than to appraise than through testing.  Why choose a particularly wasteful way to get your work done?  And if it’s just good use cases, then just call it good use cases not “test driven development.”

Today is the day I boil the ocean

Boiling the ocean.  A platitude about not, um, biting off more than you can chew… one good metaphor deserves another, right?

The oft repeated phrase around the office “don’t boil the ocean” is a warning to all those who would take on more than feasible in their improvement projects.  It’s dumb.  I mean, it’s a silly idea, after all, what would I do with a trillion tons of boiled fish?  I hardly have a fridge big enough for it all.

But more importantly, it is silly because of what people think the converse of not boiling the ocean means.  For some reason if you are not going to boil the ocean, instead you barely boil a pot of water.  After all, we wouldn’t want to get out of hand, and a large pot of salted water could easily be mistaken for the ocean right?

Ridiculousness aside, we constantly make the mistake of over focusing our improvements on such a narrow area they they hardly achieve any benefit at all.  While it might be true that boiling the ocean is a daunting if not impossible task, boiling a pot of water is so simple it has hardly any benefit to the company at all.

And so it goes with our process improvements.  If I analyze a small problem in a focused area, it takes me a couple weeks and I achieve a modest, perhaps unmeasurable benefit.  The problem is, we’re salaried employees, and almost all of our costs are labor costs, so if I don’t make an improvement that is at least as large as the value of one person’s salary (ie, thus we can eliminate said resources) then it isn’t an improvement we can realize.  I can’t cut back hours; we still pay everyone the same.  We’d have to wait around until enough of these little things add up.  And that just doesn’t make sense.

I know, I know, nobody wants to be the guy whose work is responsible for laying people off.  I admire companies that have no-layoff policies to process improvement.  It’d be great if we made cars or televisions or something where we could achieve savings in material costs.  But we have few material costs, so there is little else to eliminate but people.

My own torment about people losing their jobs aside, any project we take on that is smaller than the cost of an employee isn’t going to lead to much benefit.  Indeed, my work must additionally pay for myself as well.  At least the equivalent benefit of two salaries must be saved or else it’s a wash.

So, if you don’t want to boil the ocean, that’s fine, but at least make an attempt at boiling the contents of a large swimming pool, or even a modestly sized lake.  It just isn’t going to do you any good to boil just enough water for a cup of tea for that employee you still have to, well, employ.

It’s a silly metaphor, I realize.

Simpler can be better

We over complicate.  Almost everything.  We have advanced strategies even when we haven’t been particularly good at the basics.

Take testing for example.  We’re trying to move from being black box testers to doing risk based testing.  We’re not good at the basics of black box testing.  Far too much gets by us when we attempt to cover all the functionality.  What is going to happen when we deliberately skip some testing because we don’t think it is “risky”?  We haven’t got a clue what risky means; we don’t even know what parts of the system are error prone.

But I didn’t learn an important lesson about simpler being better from work today.  After all, it’s Sunday and I’m not at work.  But more importantly, I learned simpler is better from playing a video game.

Our friends have a Nintendo Wii.  We also have one, though we don’t own many games for it.  Our daughter never plays the Wii at home.  And furthermore, she’s just under 3 years old.  Her skill with a Wii controller is, from what we can tell, fairly limited.  But when she sees it being played at our friends’ house, she really wants to try it.

And she asks very politely to play.  In this case our friends’ son was playing Wii Playground.  It’s a series of mini-games like dodgeball, tetherball, paper airplanes, and slot cars.  My daughter has no capabilities to do anything advanced with a Wii Remote.  She can wave it about and press a button and hold it.  She has no idea about pressing a button several times, or pressing the B button.  All she can do is press the A button, and wave it about.

So our friends’ son set our daughter up with slot cars.  There are all kinds of cool things you can do with slot cars.  You can press “A” twice to make the car turbo boost, you can move the remote left and right to switch lanes, you can press “B” to activate special powers.  And doing all this, you can hop between lanes, catching speed boosts on the track and blasting in front of your opponents.

Now my little one doesn’t know about winning or losing, she just likes to try.  I’m happy for her to do that.  After all, she’s not even 3.  Anyway, once she had the remote in hand, the race started and she held down the “A” button to make the car go.

Sure enough, the car took off down the track.  Our little one didn’t move the car between lanes, she didn’t use the turbo boost, she didn’t activate any special powers.  She didn’t ram her opponents off the track.  Despite all the capabilities that this game had, she used none of them.  She just put the pedal to the metal (so to speak) by holding onto the “A” button.

She won!  She won 3 races in a row… against computer opponents, who have no idea that it’s a 2 year old playing against them!  Computer opponents don’t take it easy on you.  I’ve played the slot cars game – I have LOST at the game.  I tried the advanced strategies.  I tried to jockey for the best slot to be in, use the speed boosts, knock my opponents off the track.

Simpler was better.  My daughter, doing nothing but holding “A” beat the computer consistently.  We tend to over think, to try and use advanced techniques to get a huge edge.  We don’t need those things!  Sure, my daughter didn’t beat the opponents by a hundred car lengths, or even ten.  She just beat them.

Think simple.  Try basic.  Try boring.  Try consistent, but unexciting.  The results might surprise you.  They sure surprised the heck out of me.

Is that scalable?

Cost, quality and speed.  They’re the three things that everyone talks about as the main things to measure for an organization.  Today someone brought up one that I initially dismissed, but did have some thoughts about – scalability.  As processes go, we tend not to worry too much about scale.

What does scale mean anyway?  Well, if you ask our business partners they’ll tell you that scale is the ability to have the next transaction cost less than the prior one.  It is essentially a logarithmic curve where at some point no matter how much more you ask for the cost doesn’t go up very much.  This is a good model for a business that wants to have lots and lots of throughput from an ever increasing customer base.  The only way to get more money is to bring in more revenue which means taking on more clients at presumably an ever lower cost.

But for a software development process, I’m not sure that vision of scale is right.  Excluding funny sine-wave shaped curves, which one of these three curves represents a scalable process to you?

Choice 1: the logarithmic curve.  As the size of the request (whether it’s function points, or whatever) increases the cost very quickly rises until it caps out.  This would be OK if all you ever handled were really large requests.  You’d be efficient at doing big things, once they reached a certain “bigness”

Choice 2: the exponential curve.  As the size of the request increases the cost rises very slowly until BLAM! the cost spirals into the stratosphere.  This would be OK if all you ever handled were little things.  You’d be lean and mean, so to speak, but if someone asked for something big, you’d be incapable of doing it for a reasonable cost at all.

Choice 3: the linear relationship.  The cost of things increases in a controlled manner as the size of requests increase.  I personally think this is the right “curve” for process work and the true picture of what a scalable process means.

See, business talks about scale with this grand vision that if only we had some sort of logarithmic relationship in our process that we’d be able to get rich off each subsequent client once we had accomplished “scale.”  Having been recently reading The Black Swan, I think this would be what Mr. Taleb would call a scalable endeavour.  There’s a hope that you can get a disproportionately large reward for no more effort.  For example, the way that someone can give a concert to one person or a 50,000 person stadium and they exert the same effort either way.  The inverse would be, as Mr. Taleb likes to use, the dentist, who through the diligence of drilling a lot of teeth makes a good living, but can’t simultaneously drill a million people’s teeth.  It doesn’t “scale” which in his case means, it scales very linearly (drill one tooth get a dollar, drill 10 teeth get 10 dollars, drill 100 teeth get 100 dollars) but it doesn’t scale in some grand way.

I don’t think anyone views the exponential curve as a good thing.  However, in situations where you never did anything very large, you’d by far outpace your “logarithmic” competition.  You’d just have to recognize that you can’t take on big projects and you could be quite happy steadily costing far less than anyone else for a certain size of work.

At any rate, logarithmic scale is well and good for the business, but processes, including writing code, don’t scale like that.  One developer, working diligently turns out linearly more code for each additional amount of effort.  His work doesn’t “scale”.  An extra hour doesn’t suddenly get him 10 or 100 times the code.  So, as the size of the work requested rises, so does the cost.  Some research by Gary Gack indicates that, in fact, coding may be exponential, that we actually become less efficient at doing bigger projects.

So, when it comes to process, which “curve” do you really want?  If work of varying size comes in, which it does with software requests, then your process should be able to handle them all.  Neither the logarithmic relationship nor the exponential relationship accomplish that.  Scale for process, in my mind, means a linear relationship between the size of the request and the cost to yield that work.  Yes, that means if you ask for more work it’ll cost more, but in some linear way.  There is more “opportunity” so there is more cost.  The other two models either don’t scale up or don’t scale down.  In the linear model, process may just be able to change the slope of the line.