If you’ve not seen my talk on Performance Testing your Communcation at the Leeds Testing Atelier VIII it’s now on YouTube.
If you haven’t read it then I highly recommend you pick up a copy of The Lean Startup by Eric Ries, this was my main introduction to the term. In the book Eric talks about how a Vanity Metric is any figure which is skewed to inflate the success of a product. One of the example he gives is Number of Registered Users, a figure which will clearly go up and up over time. A better metric, Ries argues, would be Number of Active Users. Obviously if you’re looking for a successful product you’re more interested in how many users are currently using it than how many people completed the signup process.
He claims that by basing metrics and KPIs on these Vanity Metrics is akin to giving yourself a pat on the back for building a successful product while sticking your head in the sand about why your business was losing money (or the startup running out of runway as he describes it in his book).
So, to return to Tom’s question – is Velocity a vanity metric?
Most Scrum Masters calculate a teams’ velocity by taking an average over the previous sprints. The number of sprints varies but six (around 12 weeks of work) is the norm. Using this figure the the team can then calculate how far a particular story is away from delivery or epic from completion.
There’s no doubt that Velocity is a useful forecasting tool (although, as one of my colleagues pointed out recently that if you plan a sprint on your average velocity then you will, by definition, fail 50% of the time). However, is it deluding us into thinking we’re a successful team and detracting us from more accurate measurement?
I think this comes down to how you measure success. Most people would agree that judging a development team on how many features they can deliver is fairly narrow minded. A development team includes a Product Owner, their remit should be to develop a product not to simply crunch features. If that was their goal then they could simply create huge numbers of easy, yet useless features just to score points.
In this new age of DevOps a Development Team should use their PO’s expertise and take ownership of the success of their product. For them to be judged on how much work they produce, rather than how much success they’ve had is a limited measure. It reminds me of the from the book The Goal where they measured and optimised the workstations rather than asking whether the system was effective.
Therefore, although shocked when I heard it I agree with Tom. Velocity is a useful tool for the team to forecast. But if it’s chased as a KPI or used to measure the team then it is indeed a vanity metric and will distract the team from trying to improve their product. If we want to measure our teams’ success then we should look at the metrics of our products, not try to calculate some kind of Hours to Story Point ratio or chase an ever increasing Velocity. We should focus on Number of Active Users or Number of Requests via the API, this will measure the teams’ success, rather than it’s productivity. As a friend of mine is fond of saying, effective rather than efficient.
I attended Agile Yorkshire last week and saw two great talks by Tom Hoyland and Jon Fulton. I really enjoyed both but a few points in Tom’s talk really interested me and I’d like to take a few minutes to share them.
Tom is a Scrum Master at Sky Betting and Gaming, I’ve heard good things about the company in the past so I was interested to hear one of their success stories. It turns out that Tom was part of a team of twelve who really stripped Agile “back to basics” and conducted a series of experiments on the road to continuous delivery. Working in a regulated industry myself I was intrigued how they’d got on.
One of the first things Tom talked about was how many different people in the team came to the table with ideas of what was agile best practice. We all laughed at his “my guru is better than your guru” but it makes a lot of sense! I am heavily influenced by Jez Humble, Gene Kim, and Clarke Ching but many of my colleagues may watch talks and read blogs from very different thought leaders. Tom explained that one of the first thing they had to do in the team’s formation was break many of the concepts down to their fundamental concepts and understand what worked for them.
Something else Tom discussed was how the team consolidated their own backlog. This was not controversial, how else would you prioritise the work against it? It was only when he gave examples of some of the different backlogs they’re identified that I became intrigued. Risk Logs, Retro Actions, and Design Session – all of these moved onto the board and each became visible and prioritised.
It’s dangerous out there – take your buddy! I’ve heard many of the advocates or pair programming before but the idea of your buddy following you into meetings, design sessions, and CABs!? Tom explained that if your buddy went with you to these sessions not only would they learn how to design, and walk work through the CAB but they’d also know the current state of everything you were working on. If you were sick or the proverbial bus came along the team wouldn’t need to bother you because everyone would know what was going on.
There were many other good ideas (and I intend to borrow quite a few of them myself) but the final one I’m going to mention was the idea that Velocity is in fact a vanity metric (read The Lean Startup if you have no idea what I’m talking about). Velocity is just a number, like Number of Users or Number of Page Views). What we want are actionable metrics, like team predictability and accuracy of forecasts. As a Development Manager I frequently use the team’s average velocity to forecast delivery dates, Tom recommended that there are better measures out there such as a temperature check of the team’s current mood (which would often dip before any reduction in velocity). It’s an interesting idea, and one I intend to think more about over the next few weeks!
A big thank you to Royd and the guys who put Agile Yorkshire together each month. An equally big thank you to Tom and Jon for their great talks!
I have recently coined the term Lock Concept as a symptom of what many people call Fake Agile. Allow me to explain…
Waterfall development is often described with the Design, Development, and Testing phase structure. Many teams adopting Scrum tend to fall into one of two mistakes.
The first mistake is to split up these into sprints. So Sprint 1 is for design. Sprints 2, 3, and 4 are for development and testing and bug fixing will go into Sprints 5 and 6. This isn’t Scrum. Clarke Ching uses a phrase I like in his book Rolling Rocks Downhill, he talks about GETS software. That’s Good Enough To Ship, at the end of each sprint the software must be production ready. By falling into the sprint phase trap you’re lowering quality between releases and not realising the value of scrum.
The second mistake teams make is to try and run each Sprint as a mini waterfall. This is what I now describe as The Lock Complex. Teams falling into this trap will design in the first few days, develop for a few more, and then test their work towards the end. Yes, the software is GETS at the end… but doesn’t this look like a waterfall on a smaller scale?
The main symptom with this approach is people twiddling their thumbs (testers at the start of the sprint and developers at the end). While wasted time is frustrating, the real problem is the lack of shared knowledge and by unlocking that you can quickly raise your game towards Continuous Delivery.
The way to solve this becomes quite apparent if you look at the DevOps utopia we’re all told about. In a world of Continuous Delivery and automated approvals we create automated acceptance tests to ensure that our code functions as expected. If the feature doesn’t meet these automated tests then it will not be merged in, or if it has been then the deployment pipeline will stop.
In this world, not only are we deploying faster and achieving single piece flow but we’re breaking that Lock Complex. People are busy all the time and pair and mob programming becomes the norm. Instead of having a testing phase where it’s our QA’s engineers’ time to shine we have continuous collaboration and our quality specialists advising on the best tests and mechanisms to be implemented. Testers no longer run manual tests, we get computers to do that. Testers work to ensure that the automated tests give us a coherent test strategy.
If we can help our teams to break the Lock Complex and stop working in mini-waterfall sprints then we’ll see the benefits as people collaborate more and achieve better velocities and higher qualities as a result.
Our objective was to build a Death Star for the Emperor. The Dark Lord of the Sith had kindly refined the construction of his battle station into user stories and the two different teams had to build various sections ready to be deployed into production.
It was only at the end of the first sprint, after my team had constructed a very impressive throne room and firing dish that we realised that none of us had asked how we were supposed to deploy into production.
The Emperor it turns out had an operations team and was very particular about production access. We couldn’t just drop sections of the space station anywhere, there were rules, and if we didn’t talk to the operations guys, the other team, and on occasion the emperor himself there was little chance we’d succeed. I don’t want to share all the rules as IndigoBlue had clearly put a huge amount of time and effort into developing them, if you want to know more I’d highly recommend you to get in touch!
After three sprints I’m glad to say we started to get the hang of things and our Death Star reaches its MVP.
The Emperor was delighted that his Death Star had a trench surrounding it…
A board room for holding important meetings.
And a shuttle bay.
Not to mention all the other bits I didn’t get a great photo of!
Overall it was a brilliant evening, a great game to remind ourselves of some the basic Agile and DevOps concepts, and I had a lot of fun. If you see another of these events advertised I’d highly recommend it!
If you have read The Phoenix Project (which I highly recommend you do) you’ll be more than familiar with Eric who introduces us to the four types of work.
- Planned Work – these are typically business projects or new features
- Internal Projects – server migrations, software updates and so on
- Changes – usually driven by feedback on already completed work
- Unplanned Work – support escalations and emergency outages
In the book Eric talks about how Unplanned Work is unpredictable and destructive as it can often push out planned work. I can’t even begin to count how many times have I’ve planned in projects and tasks only for a crisis to arrive and my sprint to fail!
Managing a Product Backlog is an excellent way to handle three of the four types of work. After all, Agile Principles encourage us to collaborate with stakeholders, listen to feedback, and embrace change. The value of internal projects can be evaluated against other projects and then planned correspondingly and what is the backlog at all if not a prioritised list of Planned Work?
It’s the infamous Unplanned Work which continues to disrupt our Sprints, break our commitments and lower our productivity. Over a few years I’ve tried a number of different approaches to handling the inevitable Unplanned Work.
I’ve tried introducing slack into everyone’s sprint, only planning fifteen instead of twenty Story Points. I’ve tried allocating Support Tickets as they come in, assuming that the demand will cause our average sprint velocity to drop and the balance between Planned and Unplanned Work to find a happy equilibrium. It never does!
The only strategy I’ve found to work is to completely isolate the Unplanned Work from the other three types. Create a specific team (in our company we call this the SWAT team) who will react to all incoming unplanned work and “pitch in” with other work when there isn’t enough to do (Parkinson’s Law pretty much guarantee’s that this never happens).
The SWAT team handle the urgent, unplanned, and reactive work (often using a strategy like Kanban or Scrumban) while the rest of your team focus on delivering maximum value to the business (Sprints, commitments, and all the other wonderful concepts we hear about on Agile courses). That’s not to say they don’t fix bugs or take on internal projects, it’s that they work in a structured, planned environment in an effort to build solid releases and reduce the amount of Unplanned Work long term.
Speaking of Parkinson’s Law if you do decide to create this buffer between Planned and Unplanned Work (and I strongly suggest you do) you need to consider what happens when the work expands to fill the allocated time. What happens when the urgent support demands of the business exceed the capacity of your SWAT team? Hire extra people? That would be nice! But I can tell you what you must not do – DO NOT pass the extra work over into your Scrum team. I’ve made that mistake before and I can assure you that it’s a very slippery slope which is nearly impossible to come back from. Once you start roping in extra developers you undo your good work and Sprints start to fail once again. I mention it between I’ve seen it happen!
I hope I’ve given you some thoughts, I hope that my views on throttling the amount of damage your Unplanned Work can do help you safe your own commitments but I’d be delighted to hear your views – what do you to do safeguard your team’s productivity while meeting the urgent needs of the business? Post a comment below and let me know!