Independent Verifiability

As it is wont to do, the conversation about productivity has come up around the office again. Like many development shops, we fail to measure any form of productivity at all. And the primary reason we don’t measure productivity is not because we don’t track where our employees’ time goes, but because we don’t track how much work they’re doing.

See, unlike a factory which produces, say, nuts and bolts, it is difficult to count a delivered unit of functionality in software. The conventional options seem to fail us. Lines of code (LOC) is troublesome because even in an unwatched environment, the person to person variability in how many lines of code you write to get something done is quite different. Once you begin monitoring people for how many lines of code they write it encourages them to write extra lines of code, whether they are necessary for the functionality or not, in order to inflate their productivity.

The other measure commonly used, but still problematic is Function Points (FPs). FPs don’t get heavy adoption because it requires expertise to implement (plus ongoing costs) AND it turns out that developers can actually overcomplicate their design to increase the number of function points delivered – an undesirable result.

Maybe this is self evident, but the problem with both of these measures is that they aren’t independently verifiable. Instead, the counting of productivity (whether it be LOC or FPs) depends heavily on the person whose productivity you are measuring. By comparison, if you were at a factory of any kind, a layperson can count how many units come out the door. How much you produce can be verified independent of having any assistance in figuring it out.

So, what does this mean? If our existing measures of productivity can be gamed, what are we left with? Here’s my idea: we use one team in the company to measure the other. For example, we measure developer productivity in the number of test cases needed to verify all the functionality they deliver. The (perhaps big) assumption is that they developer delivers no more functionality than requested by the business, and thus the cases meant to verify that said functionality were delivered act as an independent verification of the developer. Now, since the tester is acting as the counter, you cannot measure the tester’s productivity as cost / test case executed, since that would encourage the person acting as your measurement system to want to needlessly increase the number of test cases. Indeed, we should measure testers not on cases ran but on valid defects detected, since it is their job to appraise the system and writing unnecessary test cases doesn’t add to the appraisal process.

In this way, the measurement system balances itself. The test organization measures the developers’ productivity via developer effort / test case needed to verify delivered functionality. The development organization watches over the test team because they can cancel invalid defects. Each team acts as the independent verifier of the other, thus escaping the issue of putting both the work and the reporting of how much work got done in the hands of a single team.

Is the earth still flat?

Around the year 1600 or so, Galileo had gotten himself into a great deal of trouble for his theory of heliocentrism – that the sun stands still and that the earth revolves around it. So invested we were in the bible as the only truth, that Galileo was imprisoned for his heretical beliefs. His theory was so offensive that that the church banned its publication in part or whole until 1758, over 100 years after his death.

Galileo was right, of course, as we now know. But this morning I was feeling a little like Galileo as I listened to my voicemail and a sugary-sweet (read: not so thinly veiled) message that I shouldn’t share my ideas with management without first running it by other people to make sure they agree with my opinion. What if they didn’t agree with what I was saying? Would I be banned from saying it?

Dissenting opinion is a powerful thing. If you refuse to speak the party line, are you a heretic or a hero? What if you are only vindicated after you are gone? Would you still do the right thing and offer a dissenting opinion?

In Galileo’s case, his theories held true and his work would lay the foundation for the emergence of new scientific discovery. In my case, I honestly don’t know if my opinion is a fact or simply a well educated guess based on the data I can see. I don’t sit in a position where I can change the process and see if I’m right or wrong. I’m left with only the data and what I can see by poking around. But whether I’m right or wrong isn’t the important thing. I’m actually ok with being wrong as long as it is not met with silence.

Undoubtedly we have a problem with our defect rate. That is unquestioned, but where all the defects are coming from is a question. My logical conclusion is that the defects are coming from work – either new development or bug fixing. The alternate hypothesis is that all these bugs in the code have been lying around latent for years just waiting to be discovered. Each month, more bugs than we have ever seen in a single prior month come in. Yet the established dogma says “the bugs are latent in the code.” They will not hear a dissenting opinion on the topic, which is exactly what I was offering to management.

How my story turns out is irrelevant; I don’t even know the answer yet. But it isn’t important. What is important is that dissent is not suppressed where you work. Everyone agreeing never exposes any new ideas. If we all (Galileo included) just accepted that the Sun moved around the Earth, think of all the things we might not have learned. Constructive dissent (and I do stress constructive) should be rewarded, even if the dissenter is wrong. It forces us all to consider the alternatives, to push our work in new directions, to know that what we have today is not the best we can accomplish.

Hold me, I’m scared!

Fear is a terrible thing. And while some fears are justified, they are mostly overblown. Putting your fears into perspective is difficult. My first job out of college was a contact job working on the software for a blood banking system (I’m sure I’ve written about it before). Knowing you could kill someone if you screwed up the code… that’s fear! I can see how that might paralyze you from changing the code. On the other hand, if you knew something was wrong with the project, it is the same fear that would drive you to action.

In my case, the company was developing a new test for some sort of antibody screening. As I understand it, there are several types of blood products you can give the machine that it recognizes – centrifuged whole blood, plasma, or serum. For all intents and purposes, plasma and serum are interchangeable in the types of blood tests that the system could perform. However, this new test being developed for the system only worked with plasma (or serum, I can’t remember which, but it doesn’t matter except that it only worked with one of the two blood products). However, since in the entire history of tests they were interchangeable we listed the available option as “plasma OR serum.” Now, the people using the machine were supposed to know that they should only use the right product, but the machine didn’t have any warning about it.

While I knew there was something wrong with our system, I feared telling them that they needed to delay the release of their new test was going to raise their anger. Who was I to tell them? I’m not a scientist, I’m a programmer. But I swallowed deeply and expressed my concern. Their reaction was miraculous! They listened! They stopped, looked at me, thought about it for a minute, and said “you’re right. What can we do about this?” Then they invited me (a just out-of-college programmer) to meet with their scientists and discuss a solution! The collective desire to produce the best product outweighed the fear of management’s ire that we were delaying the product while we solved this problem.

So it was with some disappointment that just this other day I encountered fear at the office again. In the business I work in, no matter what we do wrong, nobody will ever die. The risks are not nearly as great, but it is still a business and there must still be a drive to do what is best – and to be ready to hear it if we are not doing the best we can.

We were in the process of reviewing a random sample of defects which we had gone back and looked for root cause. The data was pretty detailed – people seem to remember their bugs very vividly for some reason, especially high severity bugs that cause us to back out of pilots. We took all this data and we grouped it up and looked for patterns of mistakes where we could improve the process. Lo and behold, one big thing that came out was basic coding errors – the kind that could be caught by a static analysis tool like JTest. Stupid stuff. So I said to the manager we were working with “we need to go back to development and tell them they’ve got some basic coding issues.”

At first his reaction was “isn’t that kind of cliché?” I said “why? If they have basic coding issues, they’re really there and easy to work on.”

“Isn’t it quality assurance’s fault too?” the manager said.

“When’s the last time that QA ever injected a defect into the code?” I replied. “This is a development problem and it needs to be solved by development.”

“We can’t go back to development and tell them they have sloppy programmers! They have some good programmers too!” he retorted.

Alas, fear won out over doing the right thing here. The manager I was working with feared that we couldn’t go back to development and tell them they had a basic problem. It was ok if we went to them and said “you’ve got requirements issues” or “you’ve got a knowledge gap” but for some reason, if the blame fell directly on their shoulders they were unwilling to hear “you’ve got a coding problem.” We’ll never know if they would be willing to hear it, because apparently we won’t be telling them the truth of the matter on that one. We’re afraid they might react badly.

Fear of doing the right thing: 1, Us: 0.

No good deed goes unpunished

This week was the start and end of my excursion into freecycling. Freecycling is a form of recycling where instead of taking an unused item and sending it off to the dump, you find it a new home for free. It’s a bit like selling old stuff on craigslist, but without the exchange of money. Why would anyone do this if you have something of value that could be sold? I, for one, find that selling things is a total hassle and didn’t want to hold a yard sale but didn’t want things going into the dump, and I had good stuff to get rid of. So, rather than sit all day in my yard and be haggled with or put it on craigslist, I decided to freecycle it.

In theory, freecycling has economic utility for both parties. The recipient gets a useful item and the giver gets the good feeling (yes, I do truly believe this has economic utility) of seeing their item go to good use. That was, until I actually participated in this community. Plus, in theory, once you offer things you can also request things others are offering and you essentially have a brokered trade. You earn credits by offering (though credits in this case are really an honor thing) and you redeem them by taking other stuff. It’s all very hippie, people are generally good.

Instead of having a positive freecycling experience, I had a terrible one. Rather than a few grateful people I got swarmed by apparent vultures ready to swoop down on anything of remote value. I got sob stories about why they needed my item, while in return I watched people offer stuff of relatively low value (glasses, plates, old worn out stuff with some life left but not much, coupons even). The relationship was clearly not a reciprocal one. Instead, the relationship was parasitic. Well intentioned individuals offering good things (like me) saw little opportunity to get something in return – neither good feelings nor decent trades.

So I quit the group just a few days after joining. It wasn’t worth the hassle it turned out. The problem with freecycling, so far as I can tell, is that it involves an unequal relationship. Great items, which are few and far between, disappear quickly and are swarmed by tons of interested folks. The same vultures offer relatively low value items to keep up appearances of offering as much as they take, but really aren’t putting as much skin in the game.

I’m sure I’ll generate lots of angry emails about my experience, since I’m sure people (including a friend I just chatted with) have had far more positive experiences. Anyway, I’m sure you’re wondering (given the subject of my blog) what in heck this has to do with software development. Motivation.

When we pay people to show up for work, which is what we do with salaried employees, we end up with a freecycling relationship. Because these people can take more from the company than they offer in return and goals aren’t set up to assure a mutual relationship, we get slackers. And yet, for sales people, we don’t set up this relationship. We tie their pay to sales – to actually making a difference. Why do we not do this for software developers? Why do we allow an unequal relationship?

Reviews

I don’t really know what brought this to mind – like most companies this is NOT review season, which for whatever reason appears to come in the June/July time frame. It’s also not bonus season, which often coincides with Christmas or Tax Day. But I was thinking a lot about reviews lately.

There are definitely blogs that I read which are generally opposed to reviews, and the reasons are good ones. Most commonly, reviews make performance about individual accomplishment rather than group accomplishment, which isn’t necessarily the best thing for a company. I’ve seen people counteract this with individual objectives which are actually team objectives and you either succeed or fail as a team on a measure.

The problem I have with that system is that managers often feel that they should rate the individuals “contribution” towards the shared goal rather than whether the team met the shared goal or not. In essence, managers get around the idea of true shared goals by allowing people to “succeed” even when the whole has failed.

But it wasn’t really the thing I was thinking about. What I was thinking about was the rating systems and distributions. Several companies I have been at have used the N-P-E-O scale (needs improvement, proficient, exceeds expectations, outstanding) for ratings.

The problem is, they never define what N, P, E or O means. I mean, they give a definition, but it leaves out one key element especially on the high side. If someone “exceeds expectations” does that mean that they exceed what you expect of that person or what you would expect of any person in the role? I’ve seen it used both ways, but I only agree with the latter definition. To use the first definition puts you into two problem areas: one, if you have a low performer who surprises you (in a good way) should you rate them as having exceeded expectations for that item? After all, your expectations of them were very low, so the great result is indeed exceeding your expectations. On the other hand, what if you have a high performer? If they blow you away the first time, so you rate them an outstanding, is the bar for that person now set at outstanding? Can they never impress you again?

These aren’t silly theoretical ideas… I’ve seen both examples in practice. As far as I’m concerned, you must define an expected standard for the job and each person should be rated as to whether they are below, at, or above that definition in performance. More importantly, if you don’t have a definition, then you can ascertain a definition by looking at the real distribution in capability of your existing workforce. The alternative definition applies an unequal scale, encouraging bad strategies for your employees. On one end, you have sandbaggers who keep expectations low so the can occasionally surprise you. On the other hand, you have demotivated employees who can’t ever seem to please you anymore.

Now, there’s one other aspect to this, I believe. If you haven’t defined an expectation for the job, you cannot do reviews. You can’t say that someone exceeds expectations if you have no idea what exactly the expectation is. In addition, the possibility exists, that despite there being variation person-to-person, that even the best person you have is below expected performance for the standard you set for the job. Just because a person is better than all others in the group does not automatically make them outstanding. They may be barely proficient and the rest need improvements. But just because someone blows you away compared to their peers doesn’t make them an outstanding performer.

I know that many blogs have covered this topic before, and they may not agree with me, but if I ever get back round to being in management again, having now given it several days of thought (and you think I just rattle off a blog entry whenever one comes to mind), I believe this is the model that I will adopt.

The inmates are running the asylum

I spent the other day in an all-day meeting where the team I’m on was basically asked “what business are we in?” I can see how this is a good question to ask any small company, but I’m not so sure this question applies when you are a cost center in a big company.

We exist because some other team has deemed it worthwhile to have us provide a certain set of services. It really isn’t up to us what business we want to be in. In our case, the team is there to develop and streamline processes. We were given that job in no uncertain terms. It doesn’t need to be defined anew.

So is “what business are we in?” a good question to ask your team? On one hand, reminding people that even though we are a subset of a larger business, we still need to run ourselves like a business has value. It is our job to provide valuable services and in that spirit we need to be sure we are providing competitive results – as good as, if not better, than the results that our customers would get from going elsewhere. It is good to own your own goals, to believe you helped build the processes and team.

On the other hand, if you have to ask your team what your mission is, what your vision is, and what your values are then haven’t you relinquished your leadership role in the team? After all, the manager or leader (they’re not always the same person) needs to exude the characteristics that the entire team will live by. To say to your team “what are your values?” may simply encourage inertia. Part of your job as the leader is to evaluate whether the existing culture and values are the right ones to have. If not, whatever values the team is currently living by have to change.

Case in point, at one company I worked for the team had an “always say yes” and “results are more important than the path there” attitude. This resulted in over committing and then really suffering on their way to deliver whatever it was they committed to. Not only did they not understand what their available supply was, but they didn’t care that each and every endeavor was painful. They lived day-to-day, with no thought of what the impact would be on tomorrow’s projects. As the leader of that team it was my job to come in and say “No. This isn’t in our customer’s best interests and isn’t in our own interests.” We needed a handful of core values which would not be compromised for anything.

The major value I shared with that team was “quality first.” It was unacceptable to me to deliver poor quality to our users, and I would stand behind anyone who said “I cannot deliver the quality you deserve in these circumstances.” Yes, having core values might appear to annoy your customer, but in the practice the typically short extra wait to make sure it was done right instead of just quickly was well received. Had I gone to that team and asked them what their values were, I don’t think “quality first” would have come up.

I don’t wholly object to involving your team to envision a better future for itself, but it should be done with the consideration that not everyone has that visioning capability and not everyone wants to lead. There’s nothing wrong with being a solid, capable worker. Leading is a burden as much as it is a reward, and it is understandable that not everyone wants to take it on. I’d suggest thinking twice about whether it is really the best route to let the team decide, or whether as the leader, you can model values that you want the team to have and watch them appear all on their own – no silly all-day-visioning-sessions needed.