Because I tend to focus my comments on software development processes, I subscribe to a number of newsletters just to keep abreast of stuff that other people are writing. Generally, stuff is a rehash of what we’ve already heard – see my commentary on why it seems nobody is doing primary research anymore. Today I got an interesting one which was “100 IT Performance Metrics”.
I’ve decided to rename it to “100 metrics you mostly shouldn’t bother with.” Don’t get me wrong, I love metrics and data but there are just too many issues with this proposal from Mr. Spanos to ignore.
- 100 is way too much information. It violates the idea of the critical few things you ought to control. And there are a critical few things which really make most of the difference.
- Wrong statistic. Mr. Spanos suggests the average in a lot of places where the median would likely be more appropriate. See numbers 3, 4, 9, 10, 11, etc.
- Rather than reporting one useful metric, he recommends many more than necessary. Boxplots, in many cases would better display the data than 3-4 graphs. For example, numbers 3 and 6 (mean and max resolution time) could be built into a single graph which also showed interquartile range, outliers and providing an easy to read month-over-month comparison. Just doing this could cut the number of proposed metrics in half or more.
- Stratification hell – by type, by severity. Enough said. If you solve a problem and have fewer incidents the whole number will go down. It is highly unlikely that the resolution of high severity incidents will be replaced in equal volumes by medium severity incidents. Just measure the overall number, once.
- Lack of scale. All these measures lack any sense of scale. Unlike a factory, the amount of work an IT shop is doing varies greatly from month to month. Without a normalizing factor, increases (or decreases) in any of these measures might entirely be due to changes in work volume.
- Irrelevance. Average contractor cost? Here’s a place where median is a necessity, since one expensive contractor will blow this number out of the water. Also, who cares? Are you in the business of measuring what rate the market will bear or the rate of inflation year over year? If you’ve chosen to use contractors, you’re going to have to pay for them.
- Unmeasurable. Change success rate. What the heck is a successful change? One that makes it to production? One that makes it to production with no bugs? One that yields no bugs once in production?
- Lagging. All these metrics are after-the-fact. If you live by these metrics you have to wait for bad things to happen to you before you realize it. Find some leading indicators of when you’ll be off budget or off schedule or off quality targets and use those instead.
I could go on, but why bother. This is just too easy to pick apart. If you think implementing 100 metrics is the right thing to do, you’re off your rocker. 50 is probably too many. 25 is too. Think critical few things that make your business tick. I’d guess the number is probably 3-4 output metrics and 3-4 input metrics per each output – somewhere in the range of 12 to 16 measures should get you most of the way there.
