Below Water Level

CFOs care a lot about “Waste”. Bugs in products cause more waste than what is visible to most CFOs. Paying even a little attention to bugs can change the trajectory of the business.

A 2008 Visit Etched in Memory

Back in 2008 I was working as a Software Engineer with Gluster. B.V. Jagadeesh came to visit us one day. He was the Founder and CTO of Exodus Communications, the pioneering internet company which created the high-bandwidth web hosting, application hosting and data center market.

As a young team of very young engineers, we were all very excited. Here was a technologist who had scaled the pinnacle of our field: CTO, founding a high tech company, built a great busines, leading the way for those who followed it, like AWS.

His visit came at a time when we were starting to struggle with bugs and product quality problems. GlusterFS development had been proceeding at a break-neck speed for the last few quarters. We had been optimizing for speed of getting features out. The thing we were trading-off was product quality, i.e. we were increasingly shipping a buggy product.

This was my first real job in the industry. I had a clue as to the emerging bugginess of the product and what we might try. But the serious business implication became clear because of something BV said. In one sentence B.V. Jagadeesh educated me on how to really think about bugs.

The trigger was a question from me,

“How do you think about product quality and bugs?”.

A long time has passed since the visit but some events get so deeply etched in one’s mind that you can remember the exact words and the ambience when the event took place in South End Circle, Basavanagudi.

His answer taught me something so basic yet so profound. He said:

A bug that a developer discovers in his work while coding costs 1 Dollar to fix. A bug that a QA or testing person discovers later costs 10 Dollars. But if a bug is discovered by a customer, it will cost 100 Dollars to fix.

For the naive engineer that I was back then, the idea that a bug actually costs money was unexpected. It took me a long long time to realize how critical this is for the success of any product.

True Cost of Bugs

Bugs are a kind of waste. Fixing a bug is a demand on time and money that could have been utilized elsewhere. But, that is a very simplistic understanding.

Toyota Production System outlines 8 different types of waste. All the different ways in which we tend to mis- or under-utilize the time and money at our disposal.

Applying the lens of “Waste”, it is easy to see that bugs in software result in some of the largest waste of human potential and spirit. Just by looking at the above chart, bugs cause at least 6 of the 8 types of wastes.

Type of Waste	How It Applies to Bugs
Defects	Degradation of customer experience. Delays in their work, loss of productivity, loss of value proposition.
Waiting	Customers having to wait for a fix. Customer support having to wait for fix and workarounds from developers.
Transport	Requiring movement of information between customers to support to developers and back. Additional record-keeping and collaboration costs.
Inventory	Pausing current work to go fix a bug leads to work-in-progress building up. Releases being held up for QA issues. Bug backlogs requiring constant scrubbing, grooming and prioritization.
Motion	Escalations for getting developers to fix some bugs on urgent basis, Meetings and follow-ups for ensuring threads get closed.
Extra Processing	Bug fixes requiring more PR reviews, additional testing to validate, Support to review the fix and update their own knowledge and KBs.

By decomposing waste to specific types, it becomes easy to quantify the financial implications of bugs. For eg, one can figure out the number of PRs that need to be reviewed and deployed for bug fixes. That count will help draw a connection to the amount of developer time that goes into bug fixes.

The mindset shift is to view bugs, not as a cost-of-doing-business but as a waste that can kill the profitability and agility of a business. When waste is addressed, the businesses become more agile and profitable.