The Three Ways of DevOps

If you’ve ever read The Phoenix Project (or pretty much any other DevOps book) then you’ll most have heard about the three ways.

The first time I heard the name I imagined some deep levels of understanding that the DevOps Sensai was imparting mystical knowledge to young grasshopper. I still like the mental image but having read a little deeper I believe they are three steps to building a successful collaborative team.

Let’s look at the three ways.

The First Way – Systems Thinking

The idea of the first way is to appreciate that each series of tasks is interconnected into a complex system. Recently I described how a delay at one stage in the system knocks on to each subsequent task, having a good understanding of the overall system allows you to identify where your bottlenecks are, exploit them and avoid work queuing up in front of any individual person or process.

The Second Way – Amplifying Feedback

The second way is all about looking at what happens at the end of the system and using that knowledge to improve the process. If a particular result is of a poor quality then the team need to know so they can focus more effort in the testing phases. Equally if a release is of a particularly high quality or gets good customer feedback then the team need to know that what they are doing works!

The Third Way – Experimentation

The third way is about constant experimentation and improvement. The team is willing to take risks (because they know their feedback loops are good and they will find out about any issues quickly). These teams adapt to changes quickly and are not afraid to challenge processes just because “that’s the way we’ve always done it”.
I believe that the fact these ‘ways’ are numbered is in fact very important. Without an appreciation of the end to end system it’s impossible to develop strong feedback, after all where does the system end and where should your feedback be coming from!? Equally, without this strong feedback experimentation is extremely dangerous – you may in fact be making things worse!

I also believe that these concepts can be introduced to a scrum team to lift them to the next level. By introducing ideas such as systems thinking and feedback (already a strong Agile theme) you’re asking the question of what in fact is the definition of done? DevOps encourages collaboration across Development and Operations, if a scrum team includes both roles then it’s logical that work is only “Done” when it is running properly in a customer’s environment. That surely, is the ultimate goal of any software development team?

Have you come across the The Ways? What are your thoughts? Can you use them in a scrum team?

Why Do Most Projects Finish Late?

I’m currently reading Agile Estimating and Planning by Mike Cohn, one of the things he discusses in the early chapters is why so many of projects fall behind. Many of his ideas fall in line with Eli Goldratt’s thinking in The Goal and what is described as The First Way in The Phoenix Project.

Mike discusses Parkinson’s Law which postulates that work will always take the time allocated to it. In other words if you’ve got a project and a deadline you won’t finish early because you’ll use the remaining time to refine, improve, and polish the work.

He also discusses an idea raised in The Goal where tasks are dependent on each other. In Eli Goldratt’s book Alex realises the importance of interdependencies when he takes the boy scouts walking through the woods. Cohn uses the example of developers and testers and how the person doing the QA cannot begin until the functionality has been created.

If you combine these two theories you realise that if tasks only run late, never early and the subsequent tasks can never begin until the previous one has finished you end up with an ever slipping schedule. If each task misses it’s deadline 10% of the time then once you’ve multiplied up by the number of tasks the probability that your delivery will run late climbs very quickly towards a statistical certainty!

So what can we do about this? Build in contingency? This is a risky strategy as if the team (or even you) know that there’s flexibility built into the schedule then the work will continue to consume all available time.

One approach I’ve heard a lot more recently is to allow projects to be constrained by either time or feature lists but never both. In a time based approach functionality is ranked in order of importance and when the clock runs out the project is delivered (regardless of whether or not all features are complete). In a feature driven release (for example in a lean MVP project) then the team will continue to work until all features have been completed – regardless of how long that takes.

Personally I’m much more of a fan of the first approach. By keeping this prioritised list transparent with the clients and stakeholders you can work on exactly the right work. Your work is cut when the time runs out (rather than adding low value features and running over) and one of my favourite reasons for adopting it – if a project does take less time than you expect your client gets more functionality for their money instead of feeling ripped off by inflated estimates, that’s something I’d certainly appreciate as a customer!

What do you think? How do you agree on project deadlines and commitments?

Testing Atelier

On Tuesday I was lucky enough to get tickets for one of my colleagues and I to go to the Leeds Testing Atelier. I’ve never been to one of these before but wow, the guys had worked extremely hard and created an amazing day!

There were two tracks (hipsters and nerds) throughout the day and it was action packed with different talks, topics, and workshops.

Before we got going however Clem led a group of us in a Lean Coffee session. I’ve never done one before (most likely because of my intense dislike of coffee!) but it’s definitely an idea I’ll be be trying out in team meetings at work!

I attended a couple of talks in the morning. The first was on Unit Testing best practices which I enjoyed, I got the chance to as a question on custom assertions which test multiple things (something I’ve been debating in my own head for a while). The answer by the way was “it’s ok, as long as your test continues to only test one thing” – a view I agree with!

Next up we’re a couple of short talks, one one using agile techniques to plan family life and other on website performance profiling. Both interesting and certainly talking points!

After a break Alex Carter spoke to us about the roles QAs can play in building the three ways of DevOps.

The three ways (in case you’ve not come across them are)

  1. Systems thinking
  2. Amplify feedback
  3. Continuous experimentation and improvement


It turns out that a QA is key in making this work. They’re the quality gatekeepers, they challenge processes to build quality in at all stages and act as the team’s safety net when risky changes are made. If you’ve never run through this in your head (or even better your team) then I highly recommend you do!
Lunch was pizza, in fact huge amounts of pizza! Then we headed upstairs for some QA based fun and games (some seriously difficult interviewing and spot the differences).


My final session of the day was a panel session on continuous delivery. The guys answered questions on everything from getting started to business challenges. There was a chance to ask questions at the end.

In summary the Leeds Testing Atelier was great. It was informal, informative, and had a great atmosphere with people willing to share experiences and ask questions. I’d like to thank the sponsors and organisers for all their hard work. If you’ve not been to one of these before then I’d highly recommend going to 2018 – I know I will be!

Scrum Is Not Enough!

Let me start by saying I’m a big advocate of scrum (despite some of my posts in which I challenge it over and over again). Having said that it has it’s weaknesses (like any process), one I’m going to highlight in this post is the insular nature of some scrum teams.

The best way to explain this is to describe my own experience. When I started in the Scrum Master role I was very keen on continuous delivery and wanted the development team to produce a build every two weeks which could be supplied to the business to decide whether to deploy it or not.

We had a lot of projects on at the time and we were working very hard to meet the commitments my predecessor had signed us up to and get features out the door on time.

This went on for a month or two, we hit every deadline in the calendar and provided the builds to the deployment teams on the dates we’d agreed. So what happened? Nothing…

What I’d failed to realise was that despite our hard work over the last few months we’d failed to release a single new feature to a customer. The deployment teams had struggled to install our software into UAT and without any contingency (except when it was carefully planned for) we had no capacity to assist them or pick up any issues – until the next planning session of course (where usually the next feature was the most urgent due to “customer commitments”).

Development kept on working, features continued to be produced and deadlines were hit. But the customers were sat waiting, UATs couldn’t be completed (or in some cases even installed), and the business because frustrated with us because we weren’t available to help them get the product out the door.

So what went wrong?

This is where you may have to forgive my sleight of hand in the title. I don’t believe the problem was with the scrum methodology as such, merely the most common implementations of it. The first oversight was the handover, the second was the goals of the team. Let me explain…

Firstly the handover, the issue wasn’t that it was sloppy or that it we didn’t have consistent priorities across the business (although that certainly didn’t help). The issue was that we had one… A scrum team should contain all the skills and knowledge required to get a feature from concept to customer. Rather than handing builds over to the business to deploy we should have had someone from the deployment team working in the scrum team who would actually do the implementation. The team itself would then support the new feature through it’s UAT phases and out into the customer’s live environment.

The second failing I mentioned was the team goals. I have already alluded to this but the goal of the team was to “write this feature” whereas it should have been “deliver this feature to the customer”. Only once that goal has been met should they move onto the next one.

This continuity and accountability is a very powerful thing. Projects fail when departments don’t communicate with each other or their priorities are not aligned. Systems slow when there’s too much Work in Process (for example incomplete UATs) clogging up the pipeline and generating unplanned work. If you want to break out of this cycle you need to stop thinking about departments and handovers. Stop thinking of scrum teams as groups of developers delivering feature after feature and start thinking of projects being created and delivered by teams of people from all the disciplines you need.

If you can do that, then you can make your scrum team work for the business and not only for it’s own productivity.

Measuring Sprint Velocity is Useless Unless You Ship!

I’ve recently been reading The Goal by Eli Goldratt, in it Alex Rogo (the Plant Manager) boasts to an old teacher of his that the new robots they’ve installed have greatly increased their efficiency. The Yodaesque Jonah then promptly proves his data worthless and sets Alex on the path to enlightenment.

What struck me however was how clearly this mirrors the software industry. The Goal has been the go-to book for managers for years but only with the relatively recent release of books like The Phoenix Projext have the applications to software industries been recognised.

It’s a well established idea to model a software development team like a factory. Time and money goes in, features and fixes come out. In the story Alex was delighted that his robots had given him an increased efficiency for making a particular part of the process, it was only when Jonah pointed out that the robots did not result in any increase in sales that he saw the problem in his logic.

So, let’s imagine that our software business is Alex’s factory. We’ve brought in a robot to do the work of the Development Team and Sprint Velocity has gone up from 100 Story Points to 300. As the Development Manager you get to pat yourself on the back, celebrate your success, and go home at the end of the day.

But what happens to your release at the end of your Sprint? Is your product stacked up on your factory floor awaiting the next machine (perhaps your deployment or infrastructure team)? Or have you sold it and added value to your business?

This may seem like false measurement, your numbers are telling you that you’re delivering but the reality is very different. The truth however is even worse, unshipped features are the software equivalent of Work in Progress, they’re the half finished products sat on your factory floor taking up time and space. Until your business can deliver them they’ll continue to come back and haunt you, injecting unplanned work into your Sprints and sucking time out of your Development Team.

So, If your Sprint Velocity measurements only take into account the work pushed through your Development Team then you’re not measuring the value you’re adding to the business, you’re making the same mistake as Rogo and only considering one part of the system. You may be getting great results, but are you helping achieve the goal!?

The Phoenix Project Book Review

I recently read The Phoenix Project, the definitive and often referenced DevOps Business Novel by Gene Kim, Kevin Behr, and George Spafford.

I have to admit I’ve been putting off reading this book, perhaps I had some misgivings about replacing Scrum techniques with the new craze of DevOps. Having decided that I was going to invest my time I was tested once again when I realised that Bill, the book’s main protagonist is an infrastructure guy… I’m a Development Manager, I want to know what DevOps can do for do for me not what I can do for the other guys!

That however was the last time I paused before devouring the remaining 370 pages.

Bill’s first few days after his promotion are littered with disasters. He has a project to deliver, infrastructure problems to resolve, and political ambushes to endure. In other words, he’s in the situation we’ve all experienced when teams aren’t communicating and colleagues are so stacked up with their own projects and priorities that they simply can’t assist each other.

With the timely arrival of Eric, the new potential board member Bill slowly equips himself with the tools he needs to resolve his infrastructure headaches and get the Phoenix Project back on track. Eric, acting as a kind of DevOps Yoda helps Bill gain an understanding of the four different types of work, the Theory of Constraints, Systems Thinking and see the value of Automated Deployments.

There’s a nice appendix at the end which helps consolidate the ideas and explains some of the ideas away from the story. There are also some mind blowing statistics about how some of the big web companies such as Amazon, Google, and Netflix.

So would I recommend this book to others? Absolutely! In fact I have, repeatedly to friends and colleagues! Whether you’re a developer, infrastructure guru or IT manager this book will introduce you to several new ways of thinking about workload management and IT processes.

DDDNorth 2016

I was very excited when I saw the announcement of DDDNorth earlier this year. I’ve attended the conference several times before and to see it was once again in Leeds really gave me no excuse not to attend!

Despite having a rather late night the previous evening I was one of the first there, in fact I think I accidentally tailgated a speaker inside and sat quietly in the corner until the masses arrived.

For those of you who don’t know the DDD events are free, one day conferences focused at .NET developers. They’re held around the country and have a very high standard of speakers. DDDNorth is our annual event and usually circulates between the Universities of Leeds, Sunderland and Bradford.

The first session I attended was “Machine Learning for Muggles” by Martin Kearn and it was one of the best geeky talks I have ever attended.

Martin demonstrated how to use the Azure Machine Learning API to upload your own data for analysis. He then took a photo of the audience and used facial recognition and text analysis to ask questions like “show me the happiest person”. Before today I’d always assumed machine learning was only for the likes of Google, before the first coffee break I was already wondering whether I could use the APIs in our own products.

The second session I attended was “10 More Things You Need to do to Succeed as a Tech Lead” by Joel Hammond-Turner. I’d attended one of his talks before (entitled 10 Things You Need to do to Succeed as a Tech Lead) and with my recent change in role I felt it was a wise choice session.

I was right. Joel gave us a good list. It contained tips on requirements, instrumentation, and team training. I was particularly impressed by his thoughts on managing and measuring technical debt!

The last session of the morning was “You read The Phoenix Project and you loved it! Now What?” by Matteo Emili. Having recently bought and demolished TPP while on holiday this was the session I was perhaps most excited about. Matteo argued for pragmatism, he told us that in order to sell process changes the value must be determined. He quoted the common phrase “you can’t manage what you can’t measure” and discussed in depth how you should use the Build-Measure-Learn. There were even a few tips and ticks towards the end to help us get our automation deployments going!

It was lunchtime, which meant sandwiches and GROC talks (I have always been curious to know what that stands for).

I arrived in time to see a few interesting mini-talks. The first was on using the C# interactive window in Visual Studio, the second was around ARM templates for Azure, and the final one was entitled “What’s the Point of Microsoft?” and was a tongue in cheek presentation about how the big tech players compete in today’s software and hardware world.

I’d love to continue, I’d love to tell you about the Microsoft BOT API talks I had lined up for the afternoon. Alas it was not to be, it was around this time that the office called to tell me that several hundred members were having issues with their accounts. No rest for the wicked!

I’d like to take the time to thank the DDDNorth team for putting together yet another fantastic event. The speakers this year were superb and if I get the chance I fully intend to catch the sessions I missed elsewhere in the country at a later date. See you next year!