Code Black

In total it took about 18 months to write Code Black, my recently published technical parable story. I’d originally had the idea in the summer of 2018 but it took a little time to properly outline the story.

Code Black

Instead of using a common format like The Hero’s Journey I used the various stages a team would progress through as they developed and refined their DevOps journey.

Whenever I write the first thing I do is try to outline where I want to go. This involved Mike being approached by his friend Bob (who was called Robert) at that point. Obviously he had to join the company and walk into chaos, I tried to describe a bad day we could all relate to.

As the team learns they begin to invest in more frequent releases. I wanted to explain as many of the good reasons why this was as good an idea as possible. The reduced technical risk, the reduced delivery risks, and the increased ability. I also wanted to discuss some of the common objections. Before moving onto discussing Continuous Delivery and Continuous Deployments and how using these techniques makes it less likely your sprints will fail and makes it easier to help your customer with your resorting to release branching strategies.

Once I’d outlined the story and had a basic idea of the characters it was time to sit down and write. In reality it only took a couple of months to create a first draft. Knowing where I am going always makes it a lot easier to put words in a page.

Once I’d finished writing I printed everything off and put it on a shelf for a few months. I wanted to forget as much as I could before I started proof reading so I could spot as many errors as possible.

Many of my colleagues found me over this period sat throughout lunchtime with a stack of paper and a highlighter pen. Believe me, I found a lot of things which didn’t make sense.

Once I’d corrected as much as I could it was time to publish. I’d already created my LeanPub account and in true agile style I decided it was best not to procrastinate and to start gathering feedback. The great thing about LeanPub is that it’s very easy to update your book in response to suggestions.

So that’s the story, I’ve now sold a handful of copies and so far the feedback has been very positive. I probably shouldn’t but I’m already thinking about what I should write next!

If you’re interested in picking up a copy of Code Black it’s on LeanPub now.

The Unicorn Project Book Review

When I first heard about The Unicorn Project I have to admit I was disappointed, I’ve long been a evangelist for Gene Kim’s book The Phoenix Project but I’d just spent months working on my own development DevOps book, Code Black.

I shouldn’t have worried, I really enjoyed The Unicorn Project and we’d gone down different angles. Where I’d focused more on the Continuous Deployment journey Kim’s book focuses much more on developer empowerment and continuous experimentation.

The Unicorn Project

The story follows Maxine, the developer who caused the now legendary payroll outage at Parts Unlimited towards the start of The Phoenix Project. Exiled to the documentation team as punishment she’s instructed to support the Phoenix rollout but quickly realised how woefully under supported the engineering teams are. As the business piles on more and more pressure, expects more features, and has less and less appreciation for the technical debt they’re wracking up they continue. Until, as we know the entire project explodes.

Working with some familiar characters such as Bill, Brent, Erik, and Maggie and a few new ones including Cranky Dave and Kurt our heroine works to make life better for the entire company. These are the engineers, the red-shirts, not the bridge crew. They’re the ones who actually do the work and they’ve got a lot of it to do!

What did I like best? Kim put lots of emphasis on testing and improving the entire system not just a small part of it, he focuses on collaboration and the importance of making it easy to onboard developers and share knowledge, and really drives the need to innovate and out experiment the competition. He also emphasis the importance of treating engineering tools as important systems and draws distinctions between the IT products we build, and the miscellaneous ones which just keep the lights on.

What wasn’t so good? Within a few chapters I was absolutely sick of Erik’s use of the word “sensei”… seriously can’t some of the people he quotes simply be experts, evangelists, or even gurus!?

On a more practical point the book spends a lot of time evangelising functional programming and scalability technologies. Which is great, they’re very powerful tools. But one of the things I liked so much about The Phoenix Project was how it was clear the team were struggling the same tech debt we all are. It made it more relatable and I worry in this book Kim’s “rip it out and use the latest and greatest” will overpower his more generic messages of continuous incremental improvement. Perhaps it’s personal preference but I like my DevOps books technology agnostic.

So, would I recommend this book? Absolutely without question! I believe that The Unicorn Project will take its place alongside its elder sister on the bookshelves of developers, testers, managers, product owners, and operational engineers for the next decade. If you haven’t already go and buy it, any while you’re at it not get a copy of Code Black too!? 🤣

The Lock Complex

I have recently coined the term Lock Concept as a symptom of what many people call Fake Agile. Allow me to explain…

Waterfall development is often described with the Design, Development, and Testing phase structure. Many teams adopting Scrum tend to fall into one of two mistakes.

waterfalls
Photo by Trace Hudson on Pexels.com

The first mistake is to split up these into sprints. So Sprint 1 is for design. Sprints 2, 3, and 4 are for development and testing and bug fixing will go into Sprints 5 and 6. This isn’t Scrum. Clarke Ching uses a phrase I like in his book Rolling Rocks Downhill, he talks about GETS software. That’s Good Enough To Ship, at the end of each sprint the software must be production ready. By falling into the sprint phase trap you’re lowering quality between releases and not realising the value of scrum.

The second mistake teams make is to try and run each Sprint as a mini waterfall. This is what I now describe as The Lock Complex. Teams falling into this trap will design in the first few days, develop for a few more, and then test their work towards the end. Yes, the software is GETS at the end… but doesn’t this look like a waterfall on a smaller scale?

neptune27s_staircase_2017_head-on
Canal Locks of Neptune’s Staircase by aeroid CC BY-SA 3.0

The main symptom with this approach is people twiddling their thumbs (testers at the start of the sprint and developers at the end). While wasted time is frustrating, the real problem is the lack of shared knowledge and by unlocking that you can quickly raise your game towards Continuous Delivery.

The way to solve this becomes quite apparent if you look at the DevOps utopia we’re all told about. In a world of Continuous Delivery and automated approvals we create automated acceptance tests to ensure that our code functions as expected. If the feature doesn’t meet these automated tests then it will not be merged in, or if it has been then the deployment pipeline will stop.

In this world, not only are we deploying faster and achieving single piece flow but we’re breaking that Lock Complex. People are busy all the time and pair and mob programming becomes the norm. Instead of having a testing phase where it’s our QA’s engineers’ time to shine we have continuous collaboration and our quality specialists advising on the best tests and mechanisms to be implemented. Testers no longer run manual tests, we get computers to do that. Testers work to ensure that the automated tests give us a coherent test strategy.

If we can help our teams to break the Lock Complex and stop working in mini-waterfall sprints then we’ll see the benefits as people collaborate more and achieve better velocities and higher qualities as a result.

Agile Development with Lego Star Wars

Recently I was lucky enough to attend a workshop run by IndigoBlue as part of the Leeds Digital Festival.

Our objective was to build a Death Star for the Emperor. The Dark Lord of the Sith had kindly refined the construction of his battle station into user stories and the two different teams had to build various sections ready to be deployed into production.

It was only at the end of the first sprint, after my team had constructed a very impressive throne room and firing dish that we realised that none of us had asked how we were supposed to deploy into production.

The Emperor it turns out had an operations team and was very particular about production access. We couldn’t just drop sections of the space station anywhere, there were rules, and if we didn’t talk to the operations guys, the other team, and on occasion the emperor himself there was little chance we’d succeed. I don’t want to share all the rules as IndigoBlue had clearly put a huge amount of time and effort into developing them, if you want to know more I’d highly recommend you to get in touch!

After three sprints I’m glad to say we started to get the hang of things and our Death Star reaches its MVP.

The Emperor was delighted that his Death Star had a trench surrounding it…

A board room for holding important meetings.

And a shuttle bay.

Not to mention all the other bits I didn’t get a great photo of!

Overall it was a brilliant evening, a great game to remind ourselves of some the basic Agile and DevOps concepts, and I had a lot of fun. If you see another of these events advertised I’d highly recommend it!

The Three Ways of DevOps

If you’ve ever read The Phoenix Project (or pretty much any other DevOps book) then you’ll most have heard about the three ways.

The first time I heard the name I imagined some deep levels of understanding that the DevOps Sensai was imparting mystical knowledge to young grasshopper. I still like the mental image but having read a little deeper I believe they are three steps to building a successful collaborative team.

Let’s look at the three ways.

The First Way – Systems Thinking

The idea of the first way is to appreciate that each series of tasks is interconnected into a complex system. Recently I described how a delay at one stage in the system knocks on to each subsequent task, having a good understanding of the overall system allows you to identify where your bottlenecks are, exploit them and avoid work queuing up in front of any individual person or process.

The Second Way – Amplifying Feedback

The second way is all about looking at what happens at the end of the system and using that knowledge to improve the process. If a particular result is of a poor quality then the team need to know so they can focus more effort in the testing phases. Equally if a release is of a particularly high quality or gets good customer feedback then the team need to know that what they are doing works!

The Third Way – Experimentation

The third way is about constant experimentation and improvement. The team is willing to take risks (because they know their feedback loops are good and they will find out about any issues quickly). These teams adapt to changes quickly and are not afraid to challenge processes just because “that’s the way we’ve always done it”.
I believe that the fact these ‘ways’ are numbered is in fact very important. Without an appreciation of the end to end system it’s impossible to develop strong feedback, after all where does the system end and where should your feedback be coming from!? Equally, without this strong feedback experimentation is extremely dangerous – you may in fact be making things worse!

I also believe that these concepts can be introduced to a scrum team to lift them to the next level. By introducing ideas such as systems thinking and feedback (already a strong Agile theme) you’re asking the question of what in fact is the definition of done? DevOps encourages collaboration across Development and Operations, if a scrum team includes both roles then it’s logical that work is only “Done” when it is running properly in a customer’s environment. That surely, is the ultimate goal of any software development team?

Have you come across the The Ways? What are your thoughts? Can you use them in a scrum team?

Why Do Most Projects Finish Late?

I’m currently reading Agile Estimating and Planning by Mike Cohn, one of the things he discusses in the early chapters is why so many of projects fall behind. Many of his ideas fall in line with Eli Goldratt’s thinking in The Goal and what is described as The First Way in The Phoenix Project.

Mike discusses Parkinson’s Law which postulates that work will always take the time allocated to it. In other words if you’ve got a project and a deadline you won’t finish early because you’ll use the remaining time to refine, improve, and polish the work.

He also discusses an idea raised in The Goal where tasks are dependent on each other. In Eli Goldratt’s book Alex realises the importance of interdependencies when he takes the boy scouts walking through the woods. Cohn uses the example of developers and testers and how the person doing the QA cannot begin until the functionality has been created.

If you combine these two theories you realise that if tasks only run late, never early and the subsequent tasks can never begin until the previous one has finished you end up with an ever slipping schedule. If each task misses it’s deadline 10% of the time then once you’ve multiplied up by the number of tasks the probability that your delivery will run late climbs very quickly towards a statistical certainty!

So what can we do about this? Build in contingency? This is a risky strategy as if the team (or even you) know that there’s flexibility built into the schedule then the work will continue to consume all available time.

One approach I’ve heard a lot more recently is to allow projects to be constrained by either time or feature lists but never both. In a time based approach functionality is ranked in order of importance and when the clock runs out the project is delivered (regardless of whether or not all features are complete). In a feature driven release (for example in a lean MVP project) then the team will continue to work until all features have been completed – regardless of how long that takes.

Personally I’m much more of a fan of the first approach. By keeping this prioritised list transparent with the clients and stakeholders you can work on exactly the right work. Your work is cut when the time runs out (rather than adding low value features and running over) and one of my favourite reasons for adopting it – if a project does take less time than you expect your client gets more functionality for their money instead of feeling ripped off by inflated estimates, that’s something I’d certainly appreciate as a customer!

What do you think? How do you agree on project deadlines and commitments?

Testing Atelier

On Tuesday I was lucky enough to get tickets for one of my colleagues and I to go to the Leeds Testing Atelier. I’ve never been to one of these before but wow, the guys had worked extremely hard and created an amazing day!

There were two tracks (hipsters and nerds) throughout the day and it was action packed with different talks, topics, and workshops.

Before we got going however Clem led a group of us in a Lean Coffee session. I’ve never done one before (most likely because of my intense dislike of coffee!) but it’s definitely an idea I’ll be be trying out in team meetings at work!

I attended a couple of talks in the morning. The first was on Unit Testing best practices which I enjoyed, I got the chance to as a question on custom assertions which test multiple things (something I’ve been debating in my own head for a while). The answer by the way was “it’s ok, as long as your test continues to only test one thing” – a view I agree with!

Next up we’re a couple of short talks, one one using agile techniques to plan family life and other on website performance profiling. Both interesting and certainly talking points!

After a break Alex Carter spoke to us about the roles QAs can play in building the three ways of DevOps.

The three ways (in case you’ve not come across them are)

  1. Systems thinking
  2. Amplify feedback
  3. Continuous experimentation and improvement

img_7514

It turns out that a QA is key in making this work. They’re the quality gatekeepers, they challenge processes to build quality in at all stages and act as the team’s safety net when risky changes are made. If you’ve never run through this in your head (or even better your team) then I highly recommend you do!
Lunch was pizza, in fact huge amounts of pizza! Then we headed upstairs for some QA based fun and games (some seriously difficult interviewing and spot the differences).

img_7515

My final session of the day was a panel session on continuous delivery. The guys answered questions on everything from getting started to business challenges. There was a chance to ask questions at the end.

In summary the Leeds Testing Atelier was great. It was informal, informative, and had a great atmosphere with people willing to share experiences and ask questions. I’d like to thank the sponsors and organisers for all their hard work. If you’ve not been to one of these before then I’d highly recommend going to 2018 – I know I will be!

Scrum Is Not Enough!

Let me start by saying I’m a big advocate of scrum (despite some of my posts in which I challenge it over and over again). Having said that it has it’s weaknesses (like any process), one I’m going to highlight in this post is the insular nature of some scrum teams.

The best way to explain this is to describe my own experience. When I started in the Scrum Master role I was very keen on continuous delivery and wanted the development team to produce a build every two weeks which could be supplied to the business to decide whether to deploy it or not.

We had a lot of projects on at the time and we were working very hard to meet the commitments my predecessor had signed us up to and get features out the door on time.

This went on for a month or two, we hit every deadline in the calendar and provided the builds to the deployment teams on the dates we’d agreed. So what happened? Nothing…

What I’d failed to realise was that despite our hard work over the last few months we’d failed to release a single new feature to a customer. The deployment teams had struggled to install our software into UAT and without any contingency (except when it was carefully planned for) we had no capacity to assist them or pick up any issues – until the next planning session of course (where usually the next feature was the most urgent due to “customer commitments”).

Development kept on working, features continued to be produced and deadlines were hit. But the customers were sat waiting, UATs couldn’t be completed (or in some cases even installed), and the business because frustrated with us because we weren’t available to help them get the product out the door.

So what went wrong?

This is where you may have to forgive my sleight of hand in the title. I don’t believe the problem was with the scrum methodology as such, merely the most common implementations of it. The first oversight was the handover, the second was the goals of the team. Let me explain…

Firstly the handover, the issue wasn’t that it was sloppy or that it we didn’t have consistent priorities across the business (although that certainly didn’t help). The issue was that we had one… A scrum team should contain all the skills and knowledge required to get a feature from concept to customer. Rather than handing builds over to the business to deploy we should have had someone from the deployment team working in the scrum team who would actually do the implementation. The team itself would then support the new feature through it’s UAT phases and out into the customer’s live environment.

The second failing I mentioned was the team goals. I have already alluded to this but the goal of the team was to “write this feature” whereas it should have been “deliver this feature to the customer”. Only once that goal has been met should they move onto the next one.

This continuity and accountability is a very powerful thing. Projects fail when departments don’t communicate with each other or their priorities are not aligned. Systems slow when there’s too much Work in Process (for example incomplete UATs) clogging up the pipeline and generating unplanned work. If you want to break out of this cycle you need to stop thinking about departments and handovers. Stop thinking of scrum teams as groups of developers delivering feature after feature and start thinking of projects being created and delivered by teams of people from all the disciplines you need.

If you can do that, then you can make your scrum team work for the business and not only for it’s own productivity.

Measuring Sprint Velocity is Useless Unless You Ship!

I’ve recently been reading The Goal by Eli Goldratt, in it Alex Rogo (the Plant Manager) boasts to an old teacher of his that the new robots they’ve installed have greatly increased their efficiency. The Yodaesque Jonah then promptly proves his data worthless and sets Alex on the path to enlightenment.

What struck me however was how clearly this mirrors the software industry. The Goal has been the go-to book for managers for years but only with the relatively recent release of books like The Phoenix Projext have the applications to software industries been recognised.

It’s a well established idea to model a software development team like a factory. Time and money goes in, features and fixes come out. In the story Alex was delighted that his robots had given him an increased efficiency for making a particular part of the process, it was only when Jonah pointed out that the robots did not result in any increase in sales that he saw the problem in his logic.

So, let’s imagine that our software business is Alex’s factory. We’ve brought in a robot to do the work of the Development Team and Sprint Velocity has gone up from 100 Story Points to 300. As the Development Manager you get to pat yourself on the back, celebrate your success, and go home at the end of the day.

But what happens to your release at the end of your Sprint? Is your product stacked up on your factory floor awaiting the next machine (perhaps your deployment or infrastructure team)? Or have you sold it and added value to your business?

This may seem like false measurement, your numbers are telling you that you’re delivering but the reality is very different. The truth however is even worse, unshipped features are the software equivalent of Work in Progress, they’re the half finished products sat on your factory floor taking up time and space. Until your business can deliver them they’ll continue to come back and haunt you, injecting unplanned work into your Sprints and sucking time out of your Development Team.

So, If your Sprint Velocity measurements only take into account the work pushed through your Development Team then you’re not measuring the value you’re adding to the business, you’re making the same mistake as Rogo and only considering one part of the system. You may be getting great results, but are you helping achieve the goal!?

The Phoenix Project Book Review

I recently read The Phoenix Project, the definitive and often referenced DevOps Business Novel by Gene Kim, Kevin Behr, and George Spafford.

I have to admit I’ve been putting off reading this book, perhaps I had some misgivings about replacing Scrum techniques with the new craze of DevOps. Having decided that I was going to invest my time I was tested once again when I realised that Bill, the book’s main protagonist is an infrastructure guy… I’m a Development Manager, I want to know what DevOps can do for do for me not what I can do for the other guys!

That however was the last time I paused before devouring the remaining 370 pages.

Bill’s first few days after his promotion are littered with disasters. He has a project to deliver, infrastructure problems to resolve, and political ambushes to endure. In other words, he’s in the situation we’ve all experienced when teams aren’t communicating and colleagues are so stacked up with their own projects and priorities that they simply can’t assist each other.

With the timely arrival of Eric, the new potential board member Bill slowly equips himself with the tools he needs to resolve his infrastructure headaches and get the Phoenix Project back on track. Eric, acting as a kind of DevOps Yoda helps Bill gain an understanding of the four different types of work, the Theory of Constraints, Systems Thinking and see the value of Automated Deployments.

There’s a nice appendix at the end which helps consolidate the ideas and explains some of the ideas away from the story. There are also some mind blowing statistics about how some of the big web companies such as Amazon, Google, and Netflix.

So would I recommend this book to others? Absolutely! In fact I have, repeatedly to friends and colleagues! Whether you’re a developer, infrastructure guru or IT manager this book will introduce you to several new ways of thinking about workload management and IT processes.