More information required

How long does it take to fix a defect?  No, this isn’t a “how many software developers take the screw in a light bulb” joke. 

Let’s say it’s about a month and a half from the time it is reported until closure.  That’s not so bad, right?  Anyway, a coworker was tasked with analyzing that very thing.  When Robert (just using Bob’s formal name for a change) was given the assignment he went right to work collecting data about the open and close times for defect tracking requests.

We actually keep quite a bit of data on defect resolution because we have decent work-flow management.  We can see the entire history of each request as it makes its way from submission to closure.  We can even tell how many times a request is sent back to the originator for more information.

So, Robert put together a deck he called ‘A day in the life of a bug.’  Catchy name, no?  I hate catchy names.  I view good analysis as serious business and thus deserving of a title which is serious.   When did solid analysis require a fun name?  Oh, I remember, when the analysis is inadequate.

Let’s disassemble it!

Page 1 of ‘a day in the life of a bug’:  The average duration of a request is 62 days.  Average?  Average!?!?  Not that I’m opposed to an average when it’s the appropriate measure of central tendency but this is time data we’re talking about.  Things that take time tend to be non-normal.  The reason is that you are bound on the left by zero (you can’t have things take a negative amount of time) but something can in theory go on forever.

It’s an easy test, we simply take the data and run an Anderson Darling Normality test on it.  Or, if you want to do it more graphically, we toss the data into a histogram or box plot and take a peek at it.  Well what do you know, it’s NOT normally distributed.

So, we take the median instead.  That’s about 45 days.  Wow, we’re already17 days faster than Robert claims we are.  ooops…

Page 2:  Robert provides an analysis of defect resolution by number of times a ticket is sent back to the requester for more information.  In a simple little table we get the number of times a request was sent back and the average duration of that population.  Zero times – 62 days.  One time, 65 days.  Two times, 80 days.  Three or more times, 96 days. 

Oh lord, it’s the average again… OK, the medians are:  Zero times – 42 days, One time – 45 days, Two times, 70 days, Three or more times 80 days.  Hmm, zero time or one time seems kind of close to me.  Perhaps a hypothesis test is in order.

We have non normal, but fortunately homoscedastic data.  A Kruskall-Wallis test between the two samples shows indeed that there’s no statistically significant difference between having the request never returned and having it returned once.  Beyond that, we do see statistically significant evidence that 2, 3 or more times does affect how long from request to resolution.  By the way, we call these returns to the requester “more info required”

But there’s another flaw.  See, Robert is making a case for having more controls up front for entry of the data into the system.  In theory, if the requester put in better information we’d never have to send it back for “more info required.”  It’s probably true, but on the other hand, probably not necessary.  See, logically, tickets which are sent back for “more info required” don’t take longer for a developer to resolve in terms of touch time.  The request is open longer because when it is sent back to the requester for more information they are simply lazy about getting the request updated and back into the system.

Now sure, it might improve the TAKT time for defect resolution, but it doesn’t do anything at all for the actual leanness of the resolution process.  Basically all that extra time is waiting for the request to come back around.  In the meantime the developer just works on something else.  There’s lots of issues to resolve.

So it’s fortunate that Robert produced such an analysis since it gives me something to write about and a few lessons (not that I haven’t given these before).  One: select the right measure of central tendency.  Two: check to see if there’s a statistically significant difference in the samples you are comparing.

As for Robert, we can most certainly say one thing about his analysis — “More Info Required.”  Much more info indeed.

Don’t standardize

This feels like a familiar post to me… I know that I have written in the past about the dangers of standardizing just for the sake of standardizing.  This story is a little bit different, I hope.

In a meeting about software methodology, I was told that a team out there was going to first standardize everyone onto one methodology and then look at improving it.  The logic was that there is value alone in just doing things the same way.

Now, never mind that if you don’t pick a good way of doing things you could actually be worse off than where you were before you started standardizing, but I think you might actually be destroying information.

While I agree that doing things in a standard way is important, until you know what the right standard way is, I’m not sure setting an arbitrary standard is a good move.  Even if the standard is as good as all other prior standards, it probably still isn’t a good idea.

Now for certain, it’s convenient to perform experiments on an otherwise uniform population.  For example, if you want to test a couple different fertilizers, having the same kinds of plants in both the treated and untreated groups makes sense since you eliminate variation from the plants themselves.

But if you want to explore the success of various species in their environments (which in my mind is more like exploring the effectiveness of various processes on software development) then it doesn’t make sense to eradicate all the species except one.  Once you’ve studied which are the most successful species then you can ignore the others and figure out how to improve them even more.

It seems keeping all the processes and learning from them is a better choice to me than just picking one way to succeed and ignoring the other proceses.  Don’t jump to standardize and then improve.  Instead, take the time to learn from them all, then select the best one and then improve.  Why destroy all that information you could learn from right off the bat?

Leave out the history

Today I was looking at a chart.  It was a chart about a given product line’s defects which at a glance sure looked like the product was getting significantly better over time.  The X axis read “Month 1″, “Month 2″, “Month 3″, and so on and while there was some up-and-down in the number of defects per month it mostly looked like a downward trend.

For a moment, I was fooled into thinking things really were getting better.  Then I realized that the chart only looked at the product line from a specific date, when a certain feature was rolled out.  All the history for the product prior to that point had been omitted.

On one hand, it might have been an innocent error.  Or it might have been an attempt to spin the truth.  Of course, having access to all the defect data, I went and pulled the history for this product.

Sure enough, while there did appear to be a downward trend in the recent few months, history showed that the product line had in the past produced similar downward trends, followed by upward trends and so on.  What actually was going on was nothing interesting at all.

Every time we’d introduce a new feature to the product the quality would get worse for a while while we worked out the kinks.  Then, it’d settle into a happier pattern until we introduced the next change.  So far as I could tell, this change was no different.  There was no reason to believe the quality of this change was any better than any others ever had been.

If you want to make things look good, one way you can do that is to omit the history which shows how things have really been.  After all, if you’ve got nothing to compare it to then it sure looks great, right?

How to be as difficult as possible

The purpose of having good processes is to maximize the value your customer receives.  That goes for internal customers, external customers, everyone.  So, I thought I’d share with you a story about being difficult in the name of process.

I have, as I have indiciated in prior posts, gotten my tendrils into much of the data that the organization has about process.  There are places where I have not gotten access yet.  After all, there are tons and tons of systems collecting data, acting as tools, etc.  Bringing it all together is where the magic happens.

Anyway, there was a system that I didn’t have access to which stores our test case execution data.  I had been asked to do some analysis on a claim being made by a test team.  They had argued that environment outages has impaired their ability to get their job done.  A quick and dirty check that I did showed that the number of test cases being completed in a day did not change even if a significant portion of the day was supposedly blocked by a test environment outage.

That said, I was willing to continue looking to assure I was being as thorough as possible.  One of the things I wanted to look at was the individual steps of the test case and their execution times.  Up until this point I had only been able to look at the overall test case.  In a prior post I noted that any relation between test case duration and outages seemed to be incidental.  I figured if I could get at the test steps I could control for the size of the test case and really understand if duration per step was changed during outages.

So, I started my request to the senior QA manager that I was doing all this for.  She promptly volleyed me to her immediate manager.  And in turn she gave me the name of someone who supposedly could assist me.  So I contact that person, and he promptly told me no. 

I haven’t got the energy to be nice, having already been volleyed three times, so I just send a “help!  red tape!” to the VP of the testing team.  I wasn’t going to screw around if I was trying to help their own team.  A couple messages later and the email exchange had escalated all the way up the chain on both sides.  Finally, the vice president of the test team and the support VP agreed that they’d help me out if I submitted a support request formally.

So, I submitted a support request.  I then contacted the guy who told me no in the first place and told him that the request was out there, but several hours later he told me I didn’t have a request out there.  Come to find out, someone else on his team closed the request AGAIN refusing to support my needs.  I was furious.  What was the point of this process?  Just to make it difficult to get help!?!?

Anyway, they finally reopened the request on their side and assigned it to someone who wouldn’t close it on me.  They then arranged for a phone call with me to clarify my request.  In preparation, I studied the database and did my best to author the query as close as I could get it to what needed to be run.  I was trying to make it easy for them to just get it done.

On the call, their senior manager first berated me for “causing churn” about this issue.  Irrelevant! They made it difficult to get help, I made it clear that I wasn’t taking no for an answer.  Next they asked me what exactly I needed, even though I had clearly detailed it in the email AND written the query for them.  I just couldn’t run it because I didn’t have access to the database.

Next the same person who told me no in the first place, let’s call him Fred, told me the data wouldn’t help me.  As far as I could tell it would.  He said “well, it only tells you the time the test step was completed, so if the person goes to the bathroom that’ll be included in the time.”

“I realized that, ” I replied.  “I’m assuming their amount of goofing off is relatively uniform.”

Then he said, “I can’t give you the data because you don’t have rights to it.”

I said, “I don’t want access to the database; I want you to get the data for me.  If I had access to the database I would have taken care of it myself.”

Fred clarified, “no, you don’t have rights to view that data because nobody has given you rights so I’m not going to get it for you.”

“I’ve already been granted rights through the front end.”

“Oh, ” he trailed off.  And then he continued.

“We’re not a reporting group.  This isn’t our job.”

“I understand that, and I don’t want to make a habit of this.  This is a one time request that I need for research purposes.  I don’t need it ever again.”

His manager cut in to the conversation.  “We don’t want to be viewed as difficult to work with, ” Fred’s manager said.  I’m thinking to myself, is there a way you could be MORE difficult?  “We’ve been affected by layoffs and there’s all kinds of things we haven’t been able to do.”  I felt like saying “I’m sorry to hear that.  My group of 18 people is now 4 people, so don’t give me your sob story.  I’m still doing my job.”

After a few more minutes of back and forth nonsense they agreed to get the data for me.  “It’ll take me a couple days, ” Fred said.  In reality, he had it for me that afternoon, barely 5 hours later.  Ironically, or perhaps not so surprisingly, the data he provided was in a useless PDF format.  I needed something computer readable like CSV or Excel.

Riddle me this… would it have been easier for Fred to a) argue with me over the course of two days and eventually be forced to get me the data or b) just get me the data.  Um, duh, it’s B!!!  Who designs a support process where the key feature of it is to make it as difficult as possible to get help?

I’m all for self service, if I can accomplish it.  I’m all for proper process if it makes my life easier.  But whose life was made easier by the continued refusal to help me?  It didn’t make my job easier.  In fact it wasted 2 days of my time.  It didn’t make their life easier, because if they thought that I was going to go away they were sadly mistaken.  So in turn, it wasted about a day of their time and greatly increased their aggravation levels.

Processes, whether they are customer facing or internal should serve the purpose of making getting things done easier.  That means providing mechanisms for people to self serve in an effective manner or having a process which allows you to take in and complete custom requests in an expeditious way.  It most certainly does not mean devising some horrid system to help you say NO to your customers.  Figure out how to say YES.

If you aren’t you can be rest assured that your business will be short lived.  As for this team, their services won’t be needed anymore.  I got my hands on the database access I would have rather had in the first place a day or two after this horrific interaction.  In the long term, someone else who’s more responsive will be doing their job.  For now, it’ll just be me.

Bring me problems

I have endured the ultimate insult!

One of the things we have tried to do in our organization is to remove one of the forms of waste, specifically intellectual waste.  LEAN, as far as I knew it, recognized seven forms of waste.  That was, until I saw a recent presentation about it that included an eighth form – intellectual waste.  Intellectual waste is having highly skilled workers, or specialized workers doing unintelligent work.  For example, your best sales guy being forced to data enter sales into the database rather than having an intern do that.  In the same vein, having skilled black belts hand creating charts and graphs for MBFs is waste as well.

Anyway, the company has generously, and I do truly appreciate it, invested quite a bit of time and money in me to train me as a black belt and get me certified.  But I come from a developer background, then a project manager and only most recently a process person.  I haven’t forgotten my roots; I still develop snippets of code quite often.

In fact, my ability to write code, SQL queries, and the like differentiates me from my peers in a significant manner.  While my peers are forced to utilize convenience sampling, I can select every transaction out of the database and either do proper random sampling OR take it one step further and just look at our entire history.  It also means that I can solve a form of waste for the team.  I can quickly automate queries and processing.  Alas, it is a blessing and a curse.

In turn, I have tried to share that capability with my peers by agreeing to write queries for them when necessary.  In my mind, it makes us all better to have access to better data and to not be hand-making charts instead of doing analysis.  I don’t consider programming to be intellectual waste on my part.

However, where we have gone with the programming I do consider waste.  We are a team of black belts, all of us trained and certified.  The program, like at many organizations, has been struggling and people don’t often know what to do with us.  So, often instead of problems we get requests for solutions.

People treat us as project managers and report jockeys.  They give us a thing and say “go implement this” or “report on this.”  Who hires black belts to be do-ers?  You can get a project manager anywhere.  You can get a reporting team anywhere.  This is not of value to the company to use us as shadow staff!  You have people whose full time job it is to do this!  It drives me nuts when it happens and it culminated in the ultimate insult.

Having already lowered ourselves to producing reports rather than MBFs, we had built up a set of queries which someone else wants to replicate.  Honestly, I was glad for that because if they were reporting it, we could stop.  And then we could return to something with more value.  But it wasn’t the queries themselves that made me mad, it was a passing comment about why we knew so much about the data.  “Are you guys in some sort of reporting team?”

AAAAGH!!!  I nearly lost it.  I could feel my face turn flush, so it was good that I was on the phone.  Some sort of reporting team?  Reporting team!?!?  How did we get here?  We got here because in order to do our jobs we needed to understand our measurement systems better.  People made the bad jump from “hey, they understand the transaction database” to “hey, they could produce a report faster than me making a request through normal channels.”

I will admit, as I said, a capability can be both a blessing and a curse.  And that’s what happened when I shared my development capability with the team.  The team took advantage of it and in turn shared that capability with people who just needed some reporting.  We diluted our own value by not recognizing the intellectual waste we were creating.

Bring us problems to solve, not things to do.  There are lots of less trained people who can do things for you and at least you’ll pay a fair market rate for the work too.