August 1, 2011

Incompetence or Complexity?

We recently developed a new app for a company with a sales department that is fifteen hundred reps strong. We designed the app to streamline the entire process of a sale. With a single hand-held device, a salesperson can check inventory, place an order, keep track of that order, monitor accounts payable activity, and even push marketing materials to increase the value of the account. This was a big project for us; in addition to our Silicon Valley manager, we had a team of engineers working on the project over a period of several months. One month into field-testing, the sales reps were suddenly frozen out of the app. The chief technical contact at the client called and read me the riot act: “My people can’t log on! There’s a problem with the software.” He wanted an immediate explanation. I didn’t have one.

The client demanded that we go over every bit of code we’d written and find all the bugs we’d missed. His words and his tone of voice were big yellow flags; I knew that we’d best proceed with caution. Our customer had made a common mistake. He’d taken a flying leap into a worst-case scenario, assuming that if there was one bug then there must be many. I could see he was playing the blame game and, sure enough, by the end of the conversation I was defending his implications of our incompetence. Not only did he demand we fix the problem, he wanted us to do it for free because he assumed it was our fault.

Rather than argue or defend a position, I let him blow off steam and did what I know to do: locate the technical issue rather than look for someone to blame.

Going back over every bit of code would have taken weeks. As it turned out, we found the source of the problem in a matter of hours. We had used a third-party component that was supposed to work identically across two versions, but, as it turned out did not. The glitch was an easy fix once we understood what had caused it.

In our experience, software bugs are more often the result of complexity than of developer incompetence or badly written code. More often than not bugs occur as a result of unforeseen technical issues. In other words it’s part of the process. Developing software is about writing code, testing it, evaluating test results, fixing defects, repeat as necessary. Building an effective piece of software is like any other physical, mental and emotional challenge. Over the years I’ve learned to prepare for the worst while expecting the best, and I know that even the worst can be sorted out if you keep a level head. The blame game only makes finding and fixing the problem more difficult.

The lessons are pretty obvious, but I’ll repeat them for emphasis:

  1. Resist attacking a problem by leaping to the worst case scenario. Find the actual problem.
  2. Mechanical failure or inconsistency are not equal to incompetence.
  3. Explore the most effective ways to address the failure or inconsistency.
  4. Fix the problem.