Return on Investment

I’m sure there are a fair number of places which cover what exactly a return on investment is, so I’m going to keep that part of my post brief. In short, a return on investment (ROI, hereafter) is a calculation of how long it will take you to make your money back.

For example, let’s say you spend $100 to tune up your car. Each month thereafter, you use $10 less in gasoline because your car runs more efficiently. Therefore, your ROI is $100 / $10 = 10 months until you make your money back. There are some subtleties that businesses add to that. Typically you discount future dollars saved because a dollar in the future is worth less than a dollar now.

Ultimately, the goal of doing an ROI is to decide whether to invest or not. Businesses typically look to make their money back in 3 to 5 years on an investment. The best ROIs are selected to decide where to spend your money first among many competing initiatives which want budget money. Just like in your personal life, businesses have to make decisions about what to spend your limited funds on. One alternative to any investment is always to sock the money away in some investment and just grow it. There’s no obligation to spend the money since it must always be weighed against whether it’d just be worth it to invest the cash instead.

Anyway, I digress. Having said all that, the way most businesses present an ROI is as some set of expenses divided by some amount of savings. People brainstorm what the expense factors are (labor, materials, etc.) in order to complete the project and then estimate the benefits. I don’t have any issue with the basic math that people are doing for an ROI. For the most part, people can handle the addition, subtraction, multiplication and division necessary.

Where people go wrong on an ROI is filling in the variables. Let’s take a simple example. I believe that if I create comprehensive system documentation that I can save time in design and development of new code because people won’t have to read the code to figure out how to integrate their new features. It’s pretty logical sounding, right? So, let’s say I estimate it’ll cost $100,000 to create the documentation and I estimate that it’ll save $25,000 in time…. Wait. What?

Where did my numbers come from? I made them up. And that’s what most people are doing when they do an ROI. They estimate some costs (which we tend to be more capable at doing) and then we make up some savings we’ll realize to make the ROI work. Rather than saying “well, based on past performance data projects cost us between X and Y and based on industry research we’ll probably realize between A and B savings” people just make up the numbers to make the ROI work. We already know what the goal is – an ROI in 3 to 5 years.

The problem shifts from an honest analysis of what the costs and benefits are to what numbers I need to plug in to get into that 3 to 5 year range. Microsoft Excel has a handy “goal seek” function you can even use for this. Don’t do it!

Everyone wants their idea to be invested in. Everyone can handle basic math (more or less). The problem with ROIs is all in the assumptions people make about the value that a certain variable will take on. Those values should be based on some sort of real data whether it’s neutral industry research (don’t EVER trust a company selling something), expert opinions (you should always get several opinions and account for the uncertainty), or researched data. Furthermore, you should ALWAYS be modeling the fact that results are rarely exactly always the same. Savings on a per experience basis (per project you run, for example) will vary. Many projects are not an automatic home run – there’s a good chance you’ll lose money on the deal – and you have to be honest about the possibility.

We shouldn’t be doing an ROI because we’re trying just to get past the annoyance of an investment committee. We should be doing an ROI to provide an informed view of the opportunity. We all have good ideas, but that doesn’t mean they’re the BEST idea that the company should invest in.

Beware the anecdote

“This one company used our software and they realized a $10 million savings in one year!”

Good for them. Frankly, I don’t (and you shouldn’t) care. The anecdote may be the most disastrous (from the consumer’s perspective, not the sellers) marketing device ever employed – short of outright lying I suppose. The anecdote takes a great story and uses it as the basis for why you should take your hard earned money and buy product X, or Y or Z.

Anecdotes abound. We are a culture of story tellers. The media exploits it all the time. The government exploits it all the time. Just think about the current health care debate. On one side you have the story of the Canadian woman who would have died had she relied on Canada’s healthcare. On the other side, you have a very happy Canadian health care recipient. How can it be that two totally different anecdotes are coming out of the very same system?

It’s a logical fault that we make. We assume that the related experience is the representative experience, which it clearly isn’t. Well, at least not all the time. Consider the “bad” Canadian health care experience. Let’s assume, first of all, that she really would have died if she hadn’t come to the US. We don’t actually know that because we can’t explore alternate realities. She believes she would have died, so let’s take it at face value. Has anyone in the US health care system ever died due to a bad diagnosis? If you answered “no”, I’d like to have some of whatever drug you are on, because that stuff is creating the kind of delusions not seen since the Emperor went out without his clothes on.

Of course there have been bad US experiences! People die here too! But the anecdote is powerful. It’s scary that someone might slip through the cracks. And statistics are oh so dreadfully boring! If Wikipedia is to be believed, the US pays twice as much and yet lags other wealthy nations in life expectancy and infant mortality rates. That doesn’t mean people don’t die in these other countries of horrific, potentially preventable deaths in these other countries – they do. But the odds are better. And even with improved odds of you surviving infancy and living longer, there are scary stories to tell.

People relate an anecdote when it supports their view of the world, whatever that may be. However, they’re doing that because the general patterns of the product, process or service aren’t necessarily an experience which causes you to take action.

I mean, would you be all that excited if I said “if the US adopted a health care system like Canada’s it’d probably be about the same care as you get today, but maybe a little cheaper.” Wow! I mean, wow! If that isn’t a call to action, I don’t know what is!?! People are selling things all the time – whether it is software, a change in the healthcare system, or NOT changing the health care system. If you take that anecdote and relate it to another person, who passes it along to another, and so on then it has served its powerful but potentially destructive purpose. Nobody can remember the statistics, but we all remember a good story.

The next time someone tells you a story about why you should take action, instead of listening intently, ask for the data. Then go verify the data. But most of all, beware the anecdote!

The measurement of success

How does your organization measure success? Is it your ever increasing profits? Your growing bottom line? Market share? Customer satisfaction?

Have you ever considered measuring your success as being the organization who has failed the least? Not the organization that succeeds the most, but just the one that isn’t last. As someone once said “you don’t have to be able to outrun the bear. You just have to be able to outrun the slowest guy being chased by the bear.” Still, I sat through a meeting the other day, where the praises of our “not sucking as much as everyone else” were being sung. Yes, really.

With the exception of the bear analogy, I don’t think this kind of measurement is a helpful one. Even if right now you are failing, but at the head of the pack of failures (i.e. failing the least out of all possible options), that doesn’t necessarily make you a winner. What if everyone running the marathon was measured by how far they made it before quitting rather than crossing the finish line? Nobody reaches the goal and yet we have a winner?

It just doesn’t make sense. If what you are doing isn’t meeting your customer needs, then being the closest to meeting their needs doesn’t mean your customer is happy. The space between your level of performance and what your customer actually desires, even if currently filled by no competitor in the market, is a space that can potentially be filled. And when someone fills that space, you will no longer be the guy outrunning all the rest – you’ll be the bear’s dinner.

Does anyone do primary research anymore?

I’ve been reading a lot of research lately about measurement in software development and they’re beginning to sound like a broken record. For a while I couldn’t quite put my finger on why it seemed like I was reading the same thing over and over and over again.

Finally, I figured it out. Nobody was doing any new primary research. These research papers sounded like each other because they all quoted a handful of papers who all quoted another handful of papers who all ultimately led back to just a few key pieces of research. Essentially, everything appears to have come from Capers Jones, Barry Boehm, Lister & Demarco or a handful of other places.

Everyone else’s stuff was just permutations of that. They were adding new opinions on top of the data, but no new data was being collected and analyzed. These papers lacked any statistical testing (or data for that matter) because no new research was being done. Now, maybe I’m just getting my hands on the wrong papers, but it seems far easier to find theoretical works rather than empirical works when it comes to computer science.

I find that strange, since I’m of the belief that computer science is just that, a science. It’s a science as much as any other engineering is a science. There are lots of ways to do it, but it turns out there are some best practices that just work better than others. And even though every software program we write is different, there are commonalities that allow us to generalize about situations. Somehow, unlike physics or other sciences, when someone comes out with research, nobody bothers to do anything to try and reproduce it. We just accept it at face value.

How is computer science ever supposed to advance as a science if nobody is willing to study it? We’re forever doomed to it being an art as long as we continue to treat it as one.

Get out there people and study your organizations! Build up knowledge about the science at your company because by golly we can’t count on someone else to do it for us – they’re just not doing it for some reason. Do primary research instead of relying on secondary papers which are all theoretical.