I’ve probably mentioned before that I try to read or listen to The Phoenix Project each year. Each time I pick up different things and, although rather dated now, I believe it’s still one of the most important books in our industry.
This year I’ve decided to take things a little further and run over five key revision books and listen to them back to back. My greatest hits if you like of what I feel are the cornerstones for current software development without digging into the actual code. In no particular order my books are:
The Phoenix Project
As I mentioned above TPP is one of the most important books of recent years for our industry, it’s a great introduction into devops and the idea of system thinking and should be required reading for anyone in a software development role. Lets skip over the bit where they consipire to conceal a huge customer data breach from the auditors.
The Unicorn Project
The Unicorn Project came out much more recently and tells The Phoenix Project from the perspective of Maxine, the developer who caused the payroll failure which kicked off the story. The Unicorn Project talks about the value of paying down technical debt, decoupling systems, and architecting for sustainability. It evangelises functional programming a little too much for my liking and Eric calls everyone “sensei” but it’s a very valuable and enjoyable read.
Accelerate
Ok, The Phoenix Project and The Unicorn Project are fun stories about DevOps. This is data and proof. Nicole Forsgren is a PhD, research is her day job and she was the driving force proof between The State of DevOps reports for years. In this book they delve into industry best practices and categorically prove that they lead to not only better development team satisfaction and performance but better business performance.
Rolling Rocks Downhill
You’re going to be surpised by this one but I REALLY like Rolling Rocks Downhill by Clarke Ching. It talks about many of the similar ideas of the previous three books on my list but goes into much more detail around the financial benefits and priortising options of working in an agile manner. It’s also actually really funny, I find myself chucking all the way through – something which is very rare in a technical book!
Drive
I wanted to go for something different for book number five. There were some VERY strong contenders including Team Topologies, Radical Candor, and The Lean Startup. I also can’t really list my own books… so I finally settled on Drive.
If you’ve not read any of Dan Pink’s books before they’re worth a look. He typically looks at a particular psychology idea (in this case motivation) and discusses it in nice accessible language. He’s very good at translating scientific research into business and layman’s terms.
In Drive he discusses many of the key elements which are required to keep employees motivated and happy and, shockingly many of those same aspects line up with the research conducted by Nicole Forsgren and advocated by Erik in Gene Kim’s books.
What do you think to my five revision books? What have I missed which I really must read next? I’d love to hear in the comments below or on Twitter.
This is a question I hear a lot, people have heard of (or may follow) Scrum and often have read about DevOps. They want to know whether DevOps is a replacement for Scrum or if it’s something they should be doing as well. Others believe that Scrum and DevOps are incompatible, in this post I want to talk about what both of these are how (because spoilers – they can) they should be used together.
What Is Scrum?
Lets start with the obvious question, if we’re going to discuss how DevOps and Scrum interact we need to define what exactly what we mean.
Scrum is a lightweight framework that helps people, teams and organizations generate value through adaptive solutions for complex problems.
The Scrum Guide
The Scrum Guide is an extremely concise and valuable guide to Scrum. If you haven’t read it I strongly recommend you do. However, for the purpose of this post I’m going to to highlight a few points.
Scrum employs an iterative, incremental approach to optimize predictability and to control risk.
An Increment is a concrete stepping stone toward the Product Goal. Each Increment is additive to all prior Increments and thoroughly verified, ensuring that all Increments work together. In order to provide value, the Increment must be usable.
The Definition of Done is a formal description of the state of the Increment when it meets the quality measures required for the product.
Various Sections of The Scrum Guide 2020
When many people think of Scrum they discuss the Scrum Events, these include Sprints, Retros, Planning Sessions, and Sprint Reviews. However, equal importance should be given to the underlying theory of empiricism and incremental nature.
Each Scrum Team should deliver at least one increment of work each Sprint, this increment should be potentially releasable, and meet the Definition of Done. They will then meet in a Sprint Review meeting and discuss what they should do next to deliver maximum value.
What Is DevOps?
DevOps is, at it’s core an effort to reduce the divide between development and operational teams. Literally, Dev-Ops there are many practices and ways of doing this from the cultural to the technological however what has most likely led you to this post are the ideas of Continuous Integration, Continuous Delivery, and Continuous Deployment. Again, DevOps has many valuable lessons, I’m focusing on these three not to devalue the other concepts, but simply because they are some of the key concepts we need to understand for this article.
Continuous Integration requires that each developer merges their code into the master branch every day and if the master build breaks it is resolved or rolled back within ten minutes.
Continuous Delivery is a promise that any code which is the master branch is always in a potentially deployable state. Untested and unverified code is never merged in.
Continuous Deployment is an automation mechanism which pushes whatever is in the master branch into environments (often production) without any manual steps.
As you can see these are cumulative, so teams may practice Continuous Integration but may not practice Continuous Delivery or Continuous Deployment. It is also nearly impossible to practice Continuous Deployment unless you also follow Continuous Delivery and Continuous Integration.
It’s also worth mentioning the “3 Ways of DevOps” these were popularised by the highly successful book The Phoenix Project. The 3 Ways are:
Flow/Systems Thinking
Amplify Feedback Loops
Culture of Continual Experimentation and Learning
Using Scrum and DevOps Together
Now that we’ve talked about what Scrum and DevOps are (or at least highlighted some of the relevant and key parts of each) I want to discuss whether these concepts can work in harmony together.
The first aspect to address is the Scrum concept of an increment. The Scrum Guide says that each team should produce at least one increment of “Done” software which is potentially releasable to production each sprint. I believe that the three concepts of Continuous Integration, Continuous Delivery, and Continuous Deployment are not only compatible, they are the natural evolution of this requirement.
If an engineering team is following CI, CD, and the other CD then each day (or less) a piece of work should be added to master. This means that a typical team of 7 engineers working a standard two week sprint could easily produce seventy or more increments in a single sprint. They key here is that by breaking down the Scrum Product Backlog Items down into single pieces of work which can be individually developed with a single day and then tested in isolation. This is not easy, however with practice and strong story splitting skills it can be done.
It’s also worth mentioning the Definition of Done. The Scrum Guide states that each increment must meet the Defintion of Done. Continuous Delivery states that the master branch should be always be in a deployable state. What a high performing Scrum/DevOps team should do is write automated tests which execute against incoming pull requests into their master branch to confirm that it always meets the agreed Definition of Done. This not only reduces the amount of repetitive testing work expected of the team but it highlights immediately when a proposed increment does not meet the team’s Definition of Done. If it doesn’t, it’s not merged in.
Maybe DevOps and Scrum aren’t incompatible after all? Photo by Blue Bird on Pexels.com
Before wrapping up I also want to consider the 3 Ways of DevOps I discussed above.
By building the Definition of Done into the Deployment Pipeline of the product we support the 1st Way by ensuring that the requirements of the team are baked into the pipeline.
The 2nd Way is to shorten feedback loops. Scrum emphasises the value of engaging with stakeholders frequently to ensure that they are involved and know the current state of the product. Scrum is also based on empirism, the belief that only we can only make estimates by looking at real data, by valuing the 2nd Way and shortening those feedback loops wherever possible this gives us a more accurate picture of what impact our changes have had. Simply put, this more accurate view provides the transparency we need to inspect and adapt. These are the three pillars of empirism.
Scrum also defines an event, the Retrospective where team members should meet to discuss ways to improve the quality and effectiveness of the team. This fosters the experimentation and innovation expected of the 3rd Way. These ideas aren’t working against each other. Scrum is providing events to ensure that the DevOps approaches are being honoured.
Conclusion
DevOps is often seen as a Scrum upgrade or perhaps a replacement to the agile framework. However it shouldn’t be. I believe that many of the automation and development strategies of DevOps are the natural evolution of Scrum principles and fit very neatly into any team already using the process.
With automated tests continuously testing each increment to guarantee that it meets the team’s Definition of Done increments can become smaller and a continuous flow of high value work can be delivered with shorter lead times and higher quality.
It’s been over three years since I wrote my post Four Types of Work in an Agile World and a lot has changed since then but it is still, by far the most popular post on my blog. I wanted to take a minute to revise the post as I believe things have changed, in the industry in general – not just my head!
When I wrote about this topic originally I described the four types of work introduced by The Phoenix Project as:
Planned Work
Internal Projects
Changes
Unplanned Work
I discussed how we used the product backlog to mange the first three and introduced slack into everyone’s sprint and a SWAT team to mop up as much of the Unplanned Work as possible to avoid it jeopardising the planned work.
Over the subsequent years I’ve considered this a little futher. Partially as my knowledge of DevOps and Scrum have grown, but also because – let’s face it, no one wants to be in that SWAT team!
The first change I’d make to The Phoenix Project’s famous 4 types of work is to recognise that in reality all work is either Planned or Unplanned. Internal Projects are a type of Planned Work and Changes are an implementation phase of both. Lets say instead we have:
Planned Work
Business Projects
Internal Projects
Unplanned Work
Unplanned Work will always threaten to destroy Planned Work. Photo by Christina Morillo on Pexels.com
Next I’d argue that my previous approach of having a distinct team to shelter the project team is incorrect. It’s another phase of waterfall, splitting different phases of the Develop, Verify, Run into different teams creating silos which don’t talk to each other or share learning. As someone once said – if it’s you getting paged at 3am because the website has crashed you’ll quickly make sure you resolve it in office hours!
Teams must be either orientated around projects or focused on delivering platforms as described in Team Topologies. Those teams must ensure the entire lifecycle from design to operation. To do anything else is to isolate that team from the user and reduce feedback see DevOps 2nd Way).
These teams must also ruthlessly hunt down and eliminate the technical debt which leads to this unplanned work. Creating a buffer team to try to contain the work will only last until lack of feedback and continuously declining quality consume that buffer and overflow back to the development team.
Tech Debt can only be paid down by:
Increasing the frequency of deployments (thereby decreasing the delta).
Allocating between twenty and thirty percent of the team’s capactity to preventative maintenance and tech debt work.
Improving operability and monitoring in production.
Teams have to be utterly ruthless when they resolve issues in production to take action so that they can’t happen again. Or at least that they will be much easier to detect and resolve the next time. Unfortunately, many businesses do not invest the time necessary to maintain their own products. This leads to conflict between the teams and the business and ultimately fails both sides.
This is where it is so important to have your Product Owner on side. I won’t attempt to duplicate many excellent articles on the value of paying down tech debt and having that conversation. I’d only be repeating the words of others. What I would say is that providing meaningful metrics on code quality using tools like StyleCop, Fortify, and SonarQube help quanitify the risk to the business of continuing to operate in this way.
Negotiate the composition of your backlog with your Product Owner. Photo by Tirachard Kumtanom on Pexels.com
If you want to get your Planned Work back on track you need to hunt down and destroy the sources of Unplanned Work (this is not to say you shouldn’t respond to change when it comes along). It means that Unplanned Work is a symptom of technical debt, either in the form of bugs, lack of clarity, or poor operability.
If you want your team to be able to get on with improving your product, you need to ensure that they aren’t getting distracted by people struggling to use it – otherwise you won’t have anyone wanting to use it at all!
In total it took about 18 months to write Code Black, my recently published technical parable story. I’d originally had the idea in the summer of 2018 but it took a little time to properly outline the story.
Code Black
Instead of using a common format like The Hero’s Journey I used the various stages a team would progress through as they developed and refined their DevOps journey.
Whenever I write the first thing I do is try to outline where I want to go. This involved Mike being approached by his friend Bob (who was called Robert) at that point. Obviously he had to join the company and walk into chaos, I tried to describe a bad day we could all relate to.
As the team learns they begin to invest in more frequent releases. I wanted to explain as many of the good reasons why this was as good an idea as possible. The reduced technical risk, the reduced delivery risks, and the increased ability. I also wanted to discuss some of the common objections. Before moving onto discussing Continuous Delivery and Continuous Deployments and how using these techniques makes it less likely your sprints will fail and makes it easier to help your customer with your resorting to release branching strategies.
Once I’d outlined the story and had a basic idea of the characters it was time to sit down and write. In reality it only took a couple of months to create a first draft. Knowing where I am going always makes it a lot easier to put words in a page.
Once I’d finished writing I printed everything off and put it on a shelf for a few months. I wanted to forget as much as I could before I started proof reading so I could spot as many errors as possible.
Many of my colleagues found me over this period sat throughout lunchtime with a stack of paper and a highlighter pen. Believe me, I found a lot of things which didn’t make sense.
Once I’d corrected as much as I could it was time to publish. I’d already created my LeanPub account and in true agile style I decided it was best not to procrastinate and to start gathering feedback. The great thing about LeanPub is that it’s very easy to update your book in response to suggestions.
So that’s the story, I’ve now sold a handful of copies and so far the feedback has been very positive. I probably shouldn’t but I’m already thinking about what I should write next!
If you’re interested in picking up a copy of Code Black it’s on LeanPub now.
When I first heard about The Unicorn Project I have to admit I was disappointed, I’ve long been a evangelist for Gene Kim’s book The Phoenix Project but I’d just spent months working on my own development DevOps book, Code Black.
I shouldn’t have worried, I really enjoyed The Unicorn Project and we’d gone down different angles. Where I’d focused more on the Continuous Deployment journey Kim’s book focuses much more on developer empowerment and continuous experimentation.
The Unicorn Project
The story follows Maxine, the developer who caused the now legendary payroll outage at Parts Unlimited towards the start of The Phoenix Project. Exiled to the documentation team as punishment she’s instructed to support the Phoenix rollout but quickly realised how woefully under supported the engineering teams are. As the business piles on more and more pressure, expects more features, and has less and less appreciation for the technical debt they’re wracking up they continue. Until, as we know the entire project explodes.
Working with some familiar characters such as Bill, Brent, Erik, and Maggie and a few new ones including Cranky Dave and Kurt our heroine works to make life better for the entire company. These are the engineers, the red-shirts, not the bridge crew. They’re the ones who actually do the work and they’ve got a lot of it to do!
What did I like best? Kim put lots of emphasis on testing and improving the entire system not just a small part of it, he focuses on collaboration and the importance of making it easy to onboard developers and share knowledge, and really drives the need to innovate and out experiment the competition. He also emphasis the importance of treating engineering tools as important systems and draws distinctions between the IT products we build, and the miscellaneous ones which just keep the lights on.
What wasn’t so good? Within a few chapters I was absolutely sick of Erik’s use of the word “sensei”… seriously can’t some of the people he quotes simply be experts, evangelists, or even gurus!?
On a more practical point the book spends a lot of time evangelising functional programming and scalability technologies. Which is great, they’re very powerful tools. But one of the things I liked so much about The Phoenix Project was how it was clear the team were struggling the same tech debt we all are. It made it more relatable and I worry in this book Kim’s “rip it out and use the latest and greatest” will overpower his more generic messages of continuous incremental improvement. Perhaps it’s personal preference but I like my DevOps books technology agnostic.
So, would I recommend this book? Absolutely without question! I believe that The Unicorn Project will take its place alongside its elder sister on the bookshelves of developers, testers, managers, product owners, and operational engineers for the next decade. If you haven’t already go and buy it, any while you’re at it not get a copy of Code Black too!? 🤣
I have recently coined the term Lock Concept as a symptom of what many people call Fake Agile. Allow me to explain…
Waterfall development is often described with the Design, Development, and Testing phase structure. Many teams adopting Scrum tend to fall into one of two mistakes.
The first mistake is to split up these into sprints. So Sprint 1 is for design. Sprints 2, 3, and 4 are for development and testing and bug fixing will go into Sprints 5 and 6. This isn’t Scrum. Clarke Ching uses a phrase I like in his book Rolling Rocks Downhill, he talks about GETS software. That’s Good Enough To Ship, at the end of each sprint the software must be production ready. By falling into the sprint phase trap you’re lowering quality between releases and not realising the value of scrum.
The second mistake teams make is to try and run each Sprint as a mini waterfall. This is what I now describe as The Lock Complex. Teams falling into this trap will design in the first few days, develop for a few more, and then test their work towards the end. Yes, the software is GETS at the end… but doesn’t this look like a waterfall on a smaller scale?
Canal Locks of Neptune’s Staircase by aeroid CC BY-SA 3.0
The main symptom with this approach is people twiddling their thumbs (testers at the start of the sprint and developers at the end). While wasted time is frustrating, the real problem is the lack of shared knowledge and by unlocking that you can quickly raise your game towards Continuous Delivery.
The way to solve this becomes quite apparent if you look at the DevOps utopia we’re all told about. In a world of Continuous Delivery and automated approvals we create automated acceptance tests to ensure that our code functions as expected. If the feature doesn’t meet these automated tests then it will not be merged in, or if it has been then the deployment pipeline will stop.
In this world, not only are we deploying faster and achieving single piece flow but we’re breaking that Lock Complex. People are busy all the time and pair and mob programming becomes the norm. Instead of having a testing phase where it’s our QA’s engineers’ time to shine we have continuous collaboration and our quality specialists advising on the best tests and mechanisms to be implemented. Testers no longer run manual tests, we get computers to do that. Testers work to ensure that the automated tests give us a coherent test strategy.
If we can help our teams to break the Lock Complex and stop working in mini-waterfall sprints then we’ll see the benefits as people collaborate more and achieve better velocities and higher qualities as a result.
Our objective was to build a Death Star for the Emperor. The Dark Lord of the Sith had kindly refined the construction of his battle station into user stories and the two different teams had to build various sections ready to be deployed into production.
It was only at the end of the first sprint, after my team had constructed a very impressive throne room and firing dish that we realised that none of us had asked how we were supposed to deploy into production.
The Emperor it turns out had an operations team and was very particular about production access. We couldn’t just drop sections of the space station anywhere, there were rules, and if we didn’t talk to the operations guys, the other team, and on occasion the emperor himself there was little chance we’d succeed. I don’t want to share all the rules as IndigoBlue had clearly put a huge amount of time and effort into developing them, if you want to know more I’d highly recommend you to get in touch!
After three sprints I’m glad to say we started to get the hang of things and our Death Star reaches its MVP.
The Emperor was delighted that his Death Star had a trench surrounding it…
A board room for holding important meetings.
And a shuttle bay.
Not to mention all the other bits I didn’t get a great photo of!
Overall it was a brilliant evening, a great game to remind ourselves of some the basic Agile and DevOps concepts, and I had a lot of fun. If you see another of these events advertised I’d highly recommend it!
If you’ve ever read The Phoenix Project (or pretty much any other DevOps book) then you’ll most have heard about the three ways.
The first time I heard the name I imagined some deep levels of understanding that the DevOps Sensai was imparting mystical knowledge to young grasshopper. I still like the mental image but having read a little deeper I believe they are three steps to building a successful collaborative team.
Let’s look at the three ways.
The First Way – Systems Thinking
The idea of the first way is to appreciate that each series of tasks is interconnected into a complex system. Recently I described how a delay at one stage in the system knocks on to each subsequent task, having a good understanding of the overall system allows you to identify where your bottlenecks are, exploit them and avoid work queuing up in front of any individual person or process.
The Second Way – Amplifying Feedback
The second way is all about looking at what happens at the end of the system and using that knowledge to improve the process. If a particular result is of a poor quality then the team need to know so they can focus more effort in the testing phases. Equally if a release is of a particularly high quality or gets good customer feedback then the team need to know that what they are doing works!
The Third Way – Experimentation
The third way is about constant experimentation and improvement. The team is willing to take risks (because they know their feedback loops are good and they will find out about any issues quickly). These teams adapt to changes quickly and are not afraid to challenge processes just because “that’s the way we’ve always done it”.
I believe that the fact these ‘ways’ are numbered is in fact very important. Without an appreciation of the end to end system it’s impossible to develop strong feedback, after all where does the system end and where should your feedback be coming from!? Equally, without this strong feedback experimentation is extremely dangerous – you may in fact be making things worse!
I also believe that these concepts can be introduced to a scrum team to lift them to the next level. By introducing ideas such as systems thinking and feedback (already a strong Agile theme) you’re asking the question of what in fact is the definition of done? DevOps encourages collaboration across Development and Operations, if a scrum team includes both roles then it’s logical that work is only “Done” when it is running properly in a customer’s environment. That surely, is the ultimate goal of any software development team?
Have you come across the The Ways? What are your thoughts? Can you use them in a scrum team?
I’m currently reading Agile Estimating and Planning by Mike Cohn, one of the things he discusses in the early chapters is why so many of projects fall behind. Many of his ideas fall in line with Eli Goldratt’s thinking in The Goal and what is described as The First Way in The Phoenix Project.
Mike discusses Parkinson’s Law which postulates that work will always take the time allocated to it. In other words if you’ve got a project and a deadline you won’t finish early because you’ll use the remaining time to refine, improve, and polish the work.
He also discusses an idea raised in The Goal where tasks are dependent on each other. In Eli Goldratt’s book Alex realises the importance of interdependencies when he takes the boy scouts walking through the woods. Cohn uses the example of developers and testers and how the person doing the QA cannot begin until the functionality has been created.
If you combine these two theories you realise that if tasks only run late, never early and the subsequent tasks can never begin until the previous one has finished you end up with an ever slipping schedule. If each task misses it’s deadline 10% of the time then once you’ve multiplied up by the number of tasks the probability that your delivery will run late climbs very quickly towards a statistical certainty!
So what can we do about this? Build in contingency? This is a risky strategy as if the team (or even you) know that there’s flexibility built into the schedule then the work will continue to consume all available time.
One approach I’ve heard a lot more recently is to allow projects to be constrained by either time or feature lists but never both. In a time based approach functionality is ranked in order of importance and when the clock runs out the project is delivered (regardless of whether or not all features are complete). In a feature driven release (for example in a lean MVP project) then the team will continue to work until all features have been completed – regardless of how long that takes.
Personally I’m much more of a fan of the first approach. By keeping this prioritised list transparent with the clients and stakeholders you can work on exactly the right work. Your work is cut when the time runs out (rather than adding low value features and running over) and one of my favourite reasons for adopting it – if a project does take less time than you expect your client gets more functionality for their money instead of feeling ripped off by inflated estimates, that’s something I’d certainly appreciate as a customer!
What do you think? How do you agree on project deadlines and commitments?
On Tuesday I was lucky enough to get tickets for one of my colleagues and I to go to the Leeds Testing Atelier. I’ve never been to one of these before but wow, the guys had worked extremely hard and created an amazing day!
There were two tracks (hipsters and nerds) throughout the day and it was action packed with different talks, topics, and workshops.
Before we got going however Clem led a group of us in a Lean Coffee session. I’ve never done one before (most likely because of my intense dislike of coffee!) but it’s definitely an idea I’ll be be trying out in team meetings at work!
I attended a couple of talks in the morning. The first was on Unit Testing best practices which I enjoyed, I got the chance to as a question on custom assertions which test multiple things (something I’ve been debating in my own head for a while). The answer by the way was “it’s ok, as long as your test continues to only test one thing” – a view I agree with!
Next up we’re a couple of short talks, one one using agile techniques to plan family life and other on website performance profiling. Both interesting and certainly talking points!
After a break Alex Carter spoke to us about the roles QAs can play in building the three ways of DevOps.
The three ways (in case you’ve not come across them are)
Systems thinking
Amplify feedback
Continuous experimentation and improvement
It turns out that a QA is key in making this work. They’re the quality gatekeepers, they challenge processes to build quality in at all stages and act as the team’s safety net when risky changes are made. If you’ve never run through this in your head (or even better your team) then I highly recommend you do!
Lunch was pizza, in fact huge amounts of pizza! Then we headed upstairs for some QA based fun and games (some seriously difficult interviewing and spot the differences).
My final session of the day was a panel session on continuous delivery. The guys answered questions on everything from getting started to business challenges. There was a chance to ask questions at the end.
In summary the Leeds Testing Atelier was great. It was informal, informative, and had a great atmosphere with people willing to share experiences and ask questions. I’d like to thank the sponsors and organisers for all their hard work. If you’ve not been to one of these before then I’d highly recommend going to 2018 – I know I will be!
dotdotdev.com 