The Three Ways of DevOps

If you’ve ever read The Phoenix Project (or pretty much any other DevOps book) then you’ll most have heard about the three ways.

The first time I heard the name I imagined some deep levels of understanding that the DevOps Sensai was imparting mystical knowledge to young grasshopper. I still like the mental image but having read a little deeper I believe they are three steps to building a successful collaborative team.

Let’s look at the three ways.

The First Way – Systems Thinking

The idea of the first way is to appreciate that each series of tasks is interconnected into a complex system. Recently I described how a delay at one stage in the system knocks on to each subsequent task, having a good understanding of the overall system allows you to identify where your bottlenecks are, exploit them and avoid work queuing up in front of any individual person or process.

The Second Way – Amplifying Feedback

The second way is all about looking at what happens at the end of the system and using that knowledge to improve the process. If a particular result is of a poor quality then the team need to know so they can focus more effort in the testing phases. Equally if a release is of a particularly high quality or gets good customer feedback then the team need to know that what they are doing works!

The Third Way – Experimentation

The third way is about constant experimentation and improvement. The team is willing to take risks (because they know their feedback loops are good and they will find out about any issues quickly). These teams adapt to changes quickly and are not afraid to challenge processes just because “that’s the way we’ve always done it”.
I believe that the fact these ‘ways’ are numbered is in fact very important. Without an appreciation of the end to end system it’s impossible to develop strong feedback, after all where does the system end and where should your feedback be coming from!? Equally, without this strong feedback experimentation is extremely dangerous – you may in fact be making things worse!

I also believe that these concepts can be introduced to a scrum team to lift them to the next level. By introducing ideas such as systems thinking and feedback (already a strong Agile theme) you’re asking the question of what in fact is the definition of done? DevOps encourages collaboration across Development and Operations, if a scrum team includes both roles then it’s logical that work is only “Done” when it is running properly in a customer’s environment. That surely, is the ultimate goal of any software development team?

Have you come across the The Ways? What are your thoughts? Can you use them in a scrum team?

But I’m Not In Customer Service…

I’m going to start with a confession before hopping on my soap box. I was recently copied in on a couple of emails going back and forth with a customer. I wasn’t overwhelmed by our responses, in fact there were quite a few things I would have changed, the information was accurate but it wasn’t conveyed particularly well. I was about to get involved when I had the thought “But I’m not in customer service…”

All I can say is that I’m extremely glad I didn’t say those words out loud, as soon as my internal monologue had paused for breath I realised just how little sense it was making. We are ALL in the business of customer service!

In most corporations we’re split into various teams, often with complimentary skills. We are then given goals and objectives to complete. The majority of my time is spent supporting my team and working with upcoming releases. On a day to day basis I don’t speak to our customers that frequently (unless we’re collaborating on a new feature or UAT obviously).

Does that mean I’m not in the business of customer service?

Does that mean that I don’t need to consider the customer?

Absolutely not!

Aside from the fact that it’s our customers who pay my salary as a developer the entire goal of my working day is to provide a high quality product to our customers. There are many parts of that, a good design, a high quality implementation, but also the marketing and support we provide to our customers.

If you were searching the app store and you found two equivalent 99p applications one with beautiful screenshots and another with bad grammar, typos, and a vague feature list which would you invest your money in?

If you were making the decision for a multi-million pound software investment would you rather work with the team who fob off your questions or the one one who can provide high quality answers?

Now, I’m not saying that we all need to move into marketing and join the OU’s customer service courses. But I do believe that we all need to remember that the reason our business is operating is because we have customers – providing a good service to them is in ALL of our remits from the developers to the accountants and everyone in between. Build your features with the customers’ frame of mind, write your email with the phrasing you’d like to receive and let’s not any of us mutter the phrase “but I’m not in customer service!”

Why Do Most Projects Finish Late?

I’m currently reading Agile Estimating and Planning by Mike Cohn, one of the things he discusses in the early chapters is why so many of projects fall behind. Many of his ideas fall in line with Eli Goldratt’s thinking in The Goal and what is described as The First Way in The Phoenix Project.

Mike discusses Parkinson’s Law which postulates that work will always take the time allocated to it. In other words if you’ve got a project and a deadline you won’t finish early because you’ll use the remaining time to refine, improve, and polish the work.

He also discusses an idea raised in The Goal where tasks are dependent on each other. In Eli Goldratt’s book Alex realises the importance of interdependencies when he takes the boy scouts walking through the woods. Cohn uses the example of developers and testers and how the person doing the QA cannot begin until the functionality has been created.

If you combine these two theories you realise that if tasks only run late, never early and the subsequent tasks can never begin until the previous one has finished you end up with an ever slipping schedule. If each task misses it’s deadline 10% of the time then once you’ve multiplied up by the number of tasks the probability that your delivery will run late climbs very quickly towards a statistical certainty!

So what can we do about this? Build in contingency? This is a risky strategy as if the team (or even you) know that there’s flexibility built into the schedule then the work will continue to consume all available time.

One approach I’ve heard a lot more recently is to allow projects to be constrained by either time or feature lists but never both. In a time based approach functionality is ranked in order of importance and when the clock runs out the project is delivered (regardless of whether or not all features are complete). In a feature driven release (for example in a lean MVP project) then the team will continue to work until all features have been completed – regardless of how long that takes.

Personally I’m much more of a fan of the first approach. By keeping this prioritised list transparent with the clients and stakeholders you can work on exactly the right work. Your work is cut when the time runs out (rather than adding low value features and running over) and one of my favourite reasons for adopting it – if a project does take less time than you expect your client gets more functionality for their money instead of feeling ripped off by inflated estimates, that’s something I’d certainly appreciate as a customer!

What do you think? How do you agree on project deadlines and commitments?

Using ExtensionMethods with Moq to Make Your Unit Tests More Readable

As you may have noticed from last week’s post I’ve been doing a lot of Unit Testing work recently. The product I’m working on is huge and very complex and this makes Unit Testing (as well as general coding) a real challenge. I’ve decided to write a more technical post this week focusing on a trick I’ve found which I believe really helps improve the quality of your tests.

One of the tools we’re using is Moq, a mocking framework (much like RhinoMocks or NSubstitute (one I’ve not used but comes recommended)).

One of the biggest challenges developers face when creating Unit Tests is maintaining readability as the test grows. Consider the following typical test

public void TypicalTestSetup()
  // Arrange
  var mock = new Mock<IDataAccess>();
  mock.Setup(x => x.GetIDOfRowWithGuid(It.IsAny<Guid>())).Returns(12);
  mock.Setup(x => x.GetValueFromDatabase()).Returns(1701);
  mock.Setup(x => x.SaveValue(73)).Returns(36);
  var sut = new BusinessLogic(mock.Object); 

  // Act
  var result = sut.DoWork(7);

  // Assert
  Assert.Equals(42, result);

As the mock is an object there’s no reason you can’t create ExtensionMethods for them. If I create the following method


/// <summary>
/// Sets up the method GetIDOfRowWithGuid to return the value given
/// </summary>
public static void SetupGetIDOfRowWithGuid(this Mock<IDataAccess> mock, int value)
  mock.Setup(x => x.GetIDOfRowWithGuid(It.IsAny<Guid>())).Returns(value);

 This allows us to call the setup code much more succinctly


Personally I also like to return the mock as part of the ExtensionMethod

/// <summary>
/// Sets up the method GetIDOfRowWithGuid to return the value given
/// </summary>
public static Mock<IDataAccess> SetupGetIDOfRowWithGuid(this Mock<IDataAccess> mock, int value)
  mock.Setup(x => x.GetIDOfRowWithGuid(It.IsAny<Guid>())).Returns(value);
  return mock;

Because this allows you to create more more fluent style setup instructions.

// Arrange
var mock = new Mock<IDataAccess>()
  .SetupSaveValue(73, 36);
var sut = new BusinessLogic(mock.Object);			

This, in my opinion is far more readable, simplifies your setup and can be used to create powerful, reusable setups and verifies.

Have you used ExtensionMethods with mocks? What other tips and tricks do you use to keep your Unit Tests in order?

Starting Unit Testing in a Huge Codebase

Most people agree that Unit Tests are a good idea, and most developers try to write them (with varying degrees of success). But the challenge of creating Unit Tests for existing projects can be incredibly daunting to developers.

Many people may not see the need, if an area of code has been working for quite some time then why introduce tests which will take a lot of time and (if you need to refactor to make them work) introduce risk in an area? When I wrote about the value of Unit Tests back in 2015 I postulated that the value of these small executable is not in finding bugs, it’s in preventing bugs in the future.

Unit Testing, unlike exploratory is not about finding issues with existing code, it’s about reinforcing and documenting (through executable code) exactly how each function, class, and property should behave in particular circumstances. If you understand this then you’ll see that the value of Unit Tests does not diminish with an established solution (even if most of the teething bugs have already been worked out). In fact it makes your existing code bases much easier to safely maintain!

So the question becomes how and where do we start?

I’m currently working on an application which has in excess of five million lines of code in it. Some parts are new and some date back to the project’s inception. Writing a full set of tests for the entire application is a monumental (and I have to admit largely pointless) exercise.

What we need to do is look at the areas which are most in flux. If Unit Tests are a technique for helping to protect our software against unexpected change then the areas where they deliver the most benefit are the areas of code which change frequently.

We’re engineers, not psychics (at least I’m not) so use metrics. Look at your Source Control history and see which classes are subject to frequent change (both bug fixes and feature work applies). Target your Unit Tests here, use the 80:20 rule and target your efforts in the right place. One of my favourite sayings applies here… how do you eat an elephant? A little at a time!

Slowly, little by little the classes which see the most changes will stabilise and you’ll introduce less bugs when expanding them or fixing existing defects.

These are my thoughts, do you have any experiences breaking down huge applications into Unit Tested code? How did you do it?

Scrum vs Scrumban

I wonder how many companies out there genuinely believe that they’re “Agile” because they have daily standups? I’m also curious how many people could explain the difference between Scrum and Agile? Before talking about all the merits of Scrum vs Scrumban I’m just going to reiterate a few of the basics.

Agile is a set of principles and ideas which signatories of the manifesto believe will help development of software (or actually almost businesses but for the sake of simplicity I’m going to stick to development teams).

Scrum is a process teams can follow to help them follow these principles.

Scrumban is another process which aims to uphold the Agile principles but in a slightly different way.

Most people are familiar with Scrum’s iterative approach. Software is completed every Sprint (usually every two weeks) and the upcoming work is ranked in order of priority and the team commits to delivering a certain number of those tasks in their next Sprint.

Scrumban is a little different, upcoming work is continuously reprioritised and team members pick items off the top of the list. There are no planning meetings because there’s no sprint commitment.

A few years ago we were struggling with the amount of support time our customers required. Ticket priorities were constantly changing and expecting the operations side of the business to wait for the next sprint planning session was unreasonable.

I wrote about our two team system a few weeks ago. Once we were free of trying to design a process to accommodate both Planned and Unplanned work we realised that for a ticket driven environment Scrum was no longer the approach for us.

Using Scrumban for our Support Developers means the operations team can continuously repriorise tickets in our queue as customer’s priorities change. As soon as a problem is escalated it moves to the top of the prioritised list, can you imagine the benefits after continuously breaking sprint?

We have a Kanban board in Trello, on the left are the prioritised tickets, next we have tickets a developer is actively working on, next are tickets ready to test, and finally a list of recently solved ones.

At any point I can send out the board and say “this is what we’re looking at, this is where your problem is”. Various stakeholders can negotiate with the PO to adjust priorities as required.

What’s key here is visibility, when your stakeholders can see the ordered list and can negotiate their position on it they feel far more empowered than sitting waiting for an endless queue. However in my experience it’s worth triaging this list as issues come in and letting quick win issues jump the queue to avoid long, pointless wait times.

We believe that this approach makes our Support Developers far more flexible and (hopefully) helps us provide a better customer service – that’s certainly what our numbers imply!

Have you tried Scrumban? How do you plan your Support Developers work?

What Makes Me Hire A Developer?

I’m going into a new round of interviewing developers for my team so I’ve been giving a lot of thought to what I look for when I’m interviewing people to work with.

Joel Spolsky says that you need to look for two things, brains and the ability to get things done. His reasoning is that developers who are smart but never finish jobs belong in never ending research projects, developers who get things done but aren’t smart end up causing you more work and if your candidate has neither then you should run a mile!

While I like his logic I have a few other criteria.

Joel doesn’t mention the candidate’s passion, their ability to learn, to form relationships with their team, or to empathise with the customer. In my eyes a developer who doesn’t have these qualities will negatively impact your team’s dynamic and product just as much as someone who produces bugs for a living.

So why an ability to learn? I hope this one fairly obvious. In our fast paced development world a developer’s ability to learn new technologies, absorb ideas and keep up with current trends is (in my opinion) more valuable than whether they have an in depth understanding of bitwise operators or lambda expressions. I’m not saying don’t ask about the technical side, but make sure your candidate can learn your code, your technologies and whatever comes along next!

Relationships? How many times have you worked alongside someone who can’t work in a team? They’re territorial, overly sensitive and horde knowledge with the misguided belief it makes them invulnerable. Look for people who enjoy the camaraderie and take time to teach and learn from their colleagues.

Empathy for the customer? This is, in my view one of the most important. Can your candidate put themselves in the shoes of your clients? Can they envisage how a bad release will impact your reputation? Do they understand the consequences to people’s working day when your code doesn’t work as expected? Find someone who understands the frustrations of bad software and poor customer service and you’ll find someone who will strive to prevent it!

Passion? Simply put I want a developer who wants to develop software! I don’t mean you have to spend every evening and weekend writing code or contributing to open source projects but demonstrate to me that you enjoy what you do. Tell me about what you’re working on, explain that bug fix you’re really proud of but please prove to me that you’re there’s something about the job you actually enjoy (other than just the £££s).

So there you have it, a few of the qualities I want my developer candidates to demonstrate for me. What do you look for when you’re interviewing? What personality traits do you try to show when you’re being interviewed?