The Cost of Fixing Bugs Late

Have you ever been in a situation where you’ve written a piece of code and it’s almost there but there are a few niggling issues which “can wait for V2?”. Have you ever wished, maybe months later that you’d gone back and resolved them at the time? You’re not alone!

More and more developers and managers are beginning to realise the true cost of fixing bugs late, not only on their sanity but on the company’s time and money.

Let’s say for the sake of argument that you’ve finished a piece of code except for a few edge cases. You’re under a lot of time pressure so you decide it’s of a high enough quality and you push it through to QA. The cost to the business of fixing that issue actually increases dramatically the longer it’s left.

Let’s look at a few scenarios:

You fix the issue immediatly

  • Fix time – 30 minutes

You send to QA and it’s rejected there

  • Fix time – 30 minutes
  • Build time – 2 hours
  • QA deployment time – 1 hour
  • QA signoff testing – 24 hours
  • Fix time – 30 minutes

QA passes as an “edge case” but the customer disagrees

  • Fix time – 30 minutes
  • Build time – 2 hours
  • QA deployment time – 1 hour
  • QA signoff testing – 24 hours
  • Deploy to UAT – 3 hours
  • Customer UAT – 1 week

Missed in UAT but discovered as a significant issue in Live

  • Fix time – 30 minutes
  • Build time – 2 hours
  • QA deployment time – 1 hour
  • QA signoff testing – 24 hours
  • Deploy to UAT – 3 hours
  • Customer UAT – 1 week
  • Customer live deploy – 3 hours

 

As you can see, even with these rough estimates the time it takes to resolve the customer’s issue increases dramatically the longer it’s left. Not only that but the cost of time to the business of paying staff to run extra QA cycles or rounds of UAT spirals out of control. We haven’t even considered the final case where it goes to the bug queue to die and a completely different developer has to learn the feature and resolve the problem.

The graph is actually a very common shape*.

graph-2

I’m a big fan of practicality, no software is going to be perfect and it’s unrealistic to expect that you’re going to find and fix each and every issue before shipping to a customer.

However, hopefully this has made you stop and think and has provided a strong argument for making that extra 30 minutes to resolve the issue before it causes a real headache for the business!

*Thanks to fooplot.com for the graphing software.

Stay Technical

I recently spent a Sunday night in a hotel in Hatfield prior to a customer meeting on the Monday morning. For some unknown reason I felt an urge to learn Python.

Why? I have no idea… I’m a .NET developer, my team are all .NET developers but it seemed ages since I’d picked up a new technology and had a little play… not because I’d needed to for some project but just because it looked cool!

As IT Managers and professionals we got to where we are by working very hard and having a natural curiosity about technology. Our roles may have changed but I believe it’s crucial to keep that inquisitive nature alive – especially if we want to engage with and be relevant in our teams.

Of all the managers I’ve ever worked with the ones I’ve respected the most are the ones who find the time to stay hands on. I always told myself that it was something I’d hold onto through my career.

With so much of my recent time being spent reading about the Theory of Constraints, Agile Principles, and Scrum practices… it was a refreshing change to think about immutable data types and significant white space!

The next time you have a quiet few hours do consider picking up a book or opening up pluralsight. Remind yourself why you chose to be a developer in the first place and maybe even write some code in a new language!

Agile Planning vs Planning for Agile Teams

I’ve been reading Agile Estimating and Planning by Mike Cohn recently, one of the ideas introduced in the first couple of pages is that Agile Planning is very different to planning an Agile Project.

Mike explains that as you progress throughout the project the amount of uncertainty diminishes. He calls this the Cone of Uncertainty and argues that you should continuously revise your plans as you revise your priorities.

In an agile project change is embraced and priorities are adjusted so the team are always delivering maximum value to the business. The idea of agile planning is that your plans should develop as this uncertainty reduces.

For example:

  • Sprint 1 – We estimate this is about 12-18 weeks’ worth of work
  • Sprint 2 – we’ve done some of the initial R&D and underestimated several of the user stories. We now believe the total project will take 22 weeks
  • Sprint 4 – We’re happy with our 22 week estimate at the moment but we’re getting into some of the big refactors at the moment, we’ll confirm in a few weeks
  • Sprint 6 – the majority of the work is behind us and we actually got some quick wins. We are now aiming to deliver on the 20 week mark
  • Sprint 8 – we’ve mostly finished and will be delivering on the 20th week
  • Sprint 10 – your install will be on Tuesday 15th

As you can see progress reports and updates are continuously being fed back to the stakeholders but these are updates and estimates which refine with time rather than hard deadlines before the work has even begun.

I’m very intrigued by this idea. I do have my concerns how it would work when delivering to a 3rd party – several of our customers pull large testing teams together for UAT testing of our software. I’m not sure how they’d react to a “it’ll be 12-18 weeks” but I’m certainly interested enough to continue reading!

Continuous Delivery at the Supermarket Checkout

To explain the reason for this post I should probably take a step back and explain that I’m currently fascinated with system design and the idea of workflow as described in The Phoenix Project and The Goal. I should also add (in case there’s any doubt) I hate shopping! So as Lucy and I were stood waiting in line on a particularly busy Saturday morning I had an epiphany.

Because I didn’t have anything better to do I mapped out the process in my head. The boxes on the left represent the customer, the ones on the right the cashier – at the end of the process they would tell me how much I owed and I’d dutifully hand over my money.

supermarket_flow

You’ve probably been involved in this process firsthand!

As I was loading my shopping onto the conveyor belt I couldn’t help noticing that the process wasn’t smooth. For a few minutes the belt would keep going, I was adding more and more groceries into the queue – then, for no reason I’d have to wait until more space became available.

I became convinced that if I wasn’t forced to endure this wait then the whole system would balance nicely, after all – the checkout assistant seemed to be able to scan groceries as quickly as I could load them (input and output were reasonably balanced). Then suddenly, frustratingly he would stop and complete the transaction with the customer and I had to wait for more space to become available.

In my mind the entire system mirrored a typical release schedule – features are requested, created, and released. The last part, the creation of a signed off build is often what holds up the process (either through bug fixing or code freezes) and that was exactly what I was seeing here.

Scrum practitioners advocate having a build which is good enough to ship at the end of your Sprint, this prevents large delays being caused in your process and helps make your deployments routine and safe. I’ve written previously about the quality benefits this can bring.

As I looked around me in the supermarket I began to wonder why there wasn’t a second person on the till. One could scan the items and the second would complete the transaction to ensure the workflow continued uninterrupted. That was when I realised that many have introduced something far more revolutionary!

Consider the new Scan as you Shop processes popping up around the country. These hand held devices let you scan your purchases as you work around the shop, this reduces the over complicated process above to this much simpler one:

kiosk

This simplified process reduces the need for staff and makes the entire end to end process far more efficient, there are even customer benefits such as being able to keep track of your trolley’s value as you shop. The supermarkets are so keen that they’re even willing to take financial risk on you not to steal their stock!

Tesco, by reworking their system have simplified their process and hugely increased bandwidth – I wonder if there are any similar process changes can we make in software development which will have such drastic effect our productivity?

*Thanks to draw.io for the flowchart software I used to create these images.

Scrum in a Remote Team

If you find yourself reading the Agile Manifesto (as for some reason I do from time to time) you may notice this:

The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.

I don’t think anyone would disagree with this. However in these days of satellite offices, home working, and outsourcing your team members are often scattered across cities, countries, and time zones you may not have the luxury of every team member sitting in the same room.

So what can you do?

There are a few tricks and tips I’ve found help you keep a high level of communication between team members.

  1. Ensure your daily standup times suit everyone. Make sure it doesn’t force people to start early or stay late, consider people’s lunch and prayer times. There’s no golden rule saying you must meet at 9am!
  2. Use tools like Trello to move post-it notes online.
  3. Keep a running channel open for informal chat (such as Skype or Slack) and switch important notifications onto email so people don’t miss important information.
  4. Work from home yourself, experience any pain points of your remote colleagues your are describing and aim to resolve them.
  5. Use video calling. We were a little reluctant to start this but after years of Skype the difference was noticeable, remote team members were more engaged and banter was at an all time high.

These are a few of the tips and tricks which have worked for us, what do you do to help your distributed team thrive?

Driving Technical Change Book Review

Driving Technical Change is a great little book who wants to understand how to influence different people in a technology organisation.

If you’re expecting a book on mind control then you’ll be disappointed. If you’re looking for a book which will explain why people react in particular ways to new ideas and technologies then that’s exactly what you’ll get.

Terrence Ryan discusses a series of personality caricatures. These are a little extreme, but you’ll be amazed at how closely they match up with people you know!

Then, the book discusses why these people may resist change or disagree with your proposals and how you can help to win them around.

For example, one of the characters in the book is “The Overworked”, this guy has far too much to do and never enough time. They can’t spare the time to pick up your idea, so how do you convince them? Show them how your project or proposal will save their time and help them complete their neverending pile of work.

I don’t want to plagiarise Terrence work, I’d much rather you picked up a copy and let him explain them first hand.

Would I recommend this book to someone working in a development team? Absolutely! Not just to help them drive powerful ideas forward, but also to help them appreciate other points of view – understand why The Boss, The Overworked, and The Burned may not necessarily agree with every suggestion.

Scrum Is Not Enough!

Let me start by saying I’m a big advocate of scrum (despite some of my posts in which I challenge it over and over again). Having said that it has it’s weaknesses (like any process), one I’m going to highlight in this post is the insular nature of some scrum teams.

The best way to explain this is to describe my own experience. When I started in the Scrum Master role I was very keen on continuous delivery and wanted the development team to produce a build every two weeks which could be supplied to the business to decide whether to deploy it or not.

We had a lot of projects on at the time and we were working very hard to meet the commitments my predecessor had signed us up to and get features out the door on time.

This went on for a month or two, we hit every deadline in the calendar and provided the builds to the deployment teams on the dates we’d agreed. So what happened? Nothing…

What I’d failed to realise was that despite our hard work over the last few months we’d failed to release a single new feature to a customer. The deployment teams had struggled to install our software into UAT and without any contingency (except when it was carefully planned for) we had no capacity to assist them or pick up any issues – until the next planning session of course (where usually the next feature was the most urgent due to “customer commitments”).

Development kept on working, features continued to be produced and deadlines were hit. But the customers were sat waiting, UATs couldn’t be completed (or in some cases even installed), and the business because frustrated with us because we weren’t available to help them get the product out the door.

So what went wrong?

This is where you may have to forgive my sleight of hand in the title. I don’t believe the problem was with the scrum methodology as such, merely the most common implementations of it. The first oversight was the handover, the second was the goals of the team. Let me explain…

Firstly the handover, the issue wasn’t that it was sloppy or that it we didn’t have consistent priorities across the business (although that certainly didn’t help). The issue was that we had one… A scrum team should contain all the skills and knowledge required to get a feature from concept to customer. Rather than handing builds over to the business to deploy we should have had someone from the deployment team working in the scrum team who would actually do the implementation. The team itself would then support the new feature through it’s UAT phases and out into the customer’s live environment.

The second failing I mentioned was the team goals. I have already alluded to this but the goal of the team was to “write this feature” whereas it should have been “deliver this feature to the customer”. Only once that goal has been met should they move onto the next one.

This continuity and accountability is a very powerful thing. Projects fail when departments don’t communicate with each other or their priorities are not aligned. Systems slow when there’s too much Work in Process (for example incomplete UATs) clogging up the pipeline and generating unplanned work. If you want to break out of this cycle you need to stop thinking about departments and handovers. Stop thinking of scrum teams as groups of developers delivering feature after feature and start thinking of projects being created and delivered by teams of people from all the disciplines you need.

If you can do that, then you can make your scrum team work for the business and not only for it’s own productivity.