Testing Atelier

On Tuesday I was lucky enough to get tickets for one of my colleagues and I to go to the Leeds Testing Atelier. I’ve never been to one of these before but wow, the guys had worked extremely hard and created an amazing day!

There were two tracks (hipsters and nerds) throughout the day and it was action packed with different talks, topics, and workshops.

Before we got going however Clem led a group of us in a Lean Coffee session. I’ve never done one before (most likely because of my intense dislike of coffee!) but it’s definitely an idea I’ll be be trying out in team meetings at work!

I attended a couple of talks in the morning. The first was on Unit Testing best practices which I enjoyed, I got the chance to as a question on custom assertions which test multiple things (something I’ve been debating in my own head for a while). The answer by the way was “it’s ok, as long as your test continues to only test one thing” – a view I agree with!

Next up we’re a couple of short talks, one one using agile techniques to plan family life and other on website performance profiling. Both interesting and certainly talking points!

After a break Alex Carter spoke to us about the roles QAs can play in building the three ways of DevOps.

The three ways (in case you’ve not come across them are)

  1. Systems thinking
  2. Amplify feedback
  3. Continuous experimentation and improvement


It turns out that a QA is key in making this work. They’re the quality gatekeepers, they challenge processes to build quality in at all stages and act as the team’s safety net when risky changes are made. If you’ve never run through this in your head (or even better your team) then I highly recommend you do!
Lunch was pizza, in fact huge amounts of pizza! Then we headed upstairs for some QA based fun and games (some seriously difficult interviewing and spot the differences).


My final session of the day was a panel session on continuous delivery. The guys answered questions on everything from getting started to business challenges. There was a chance to ask questions at the end.

In summary the Leeds Testing Atelier was great. It was informal, informative, and had a great atmosphere with people willing to share experiences and ask questions. I’d like to thank the sponsors and organisers for all their hard work. If you’ve not been to one of these before then I’d highly recommend going to 2018 – I know I will be!

Have you tried At Desk Testing?

Last week I wrote about the value of finding issues early. How it becomes increasingly expensive and time consuming to fix issues the further down the development lifecycle you get. With that in mind we can now appreciate that anything we can do to find bugs earlier makes our software not only better but cheaper to develop.

Something we’re trialling at the moment are at desk demos. The idea is simple, before signing off a piece of work and passing it onto the next link in the chain (Dev to QA, QA to Support Analyst, Support Analyst to Dev and so on) you demonstrate the issue or feature to them.

For example, before I finish a feature and pass it onto someone who specialises in testing I invite my buddy over to “give it a bash”.

Remember last week? I talked about the time it takes to move from on link in this chain to another. I discussed how it can take a few hours to build your software, another hour or so to deploy it, and a day to run the signoff scripts (obviously this varies if you’re fortunate enough to be working on a ‘modern’ solution or have invested in some proper CI). Time moving backwards is time wasted, if you can avoid rework then you should always take the opportunity to do so!
By offering up my work to the QA for a few minutes before formally handing over can save hours of wasted time. These guys know what they’re looking for and can often find edge cases and give feedback on a few scenarios you’ve not considered. By having these pointed out to you early you’re saving all this extra time!

The same theory can be applied to a Support Analyst demoing bugs to a Developer rather than just recording replication steps on a ticket or a Developer showing a bug fix to the same analyst before shipping it to a customer’s UAT environment for testing.

So far it’s working well for us. Do you demo before handing over? Do you feel it works for your team?

When and Where to Automate Testing

A year ago I undertook an interesting piece of R&D to write Selenium tests for our main UI. I watched the pluralsight course, learned the difference between WebDriver and the IDE, and started building my Page Object Model. My simple test took the best part of three weeks to build and executed in around half the time a decent QA would take if you gave them a double shot of espresso.

I patted myself on the back, demonstrated the work to our Product Owner, and then advised that we should shelve the work because I wasn’t confident to hand over the mind boggling complexities of waits, framesets, and inherited view models into general practice.

Over time I lost confidence in the project. If it had taken me days to generate even the simplest of tests how would a junior developer fair when asked to automate complex financial scenarios. Quietly I put the idea to the back of my mind and concentrated on other more pressing matters.

That was until last week. We’ve been doing some work on our BACS integration and as part of the regression test I’d enlisted the help of one of our QAs to mock each response code and import it into the system. The process was tedious, repetitive, and I hated myself when I had to tell him he’d missed a vital check off each scenario.

As I was speaking to him cogs in my head began to turn. I’d gone off the idea of large scale test automation because of the complexity of our UI but the BACS processing system doesn’t have a UI. I could knock something together in a couple of hours which would create customers, mock BACS files, and schedule our JobServer. Even more powerful, if I used a technology like SpecFlow our QA could write the tests he wanted, I could automate them, and we’d be able to iterate over every scenario within a couple of minutes. Even more exciting was the idea we could send the feature files off to our Product Owner and banking partner and ask them to verify the behaviour was correct.

Later that week, after we’d proven our project to be a success and found a handful of low priority bugs to be corrected I started to wonder why this automation project had delivered such value where the previous UI automation had failed. I decided this was because:

  • The BACS process was an automated mechanism already so the test automation steps were simpler
  • The UI had been designed for human use, it wasn’t a chore to run through but it was complex for Selenium to navigate the controls
  • Mocking and importing BACS files was repetitive, slow, and tedious

The project turned out to be a huge success, we’re already planning on how we can expand the solution to cover other payment integrations such as SEPA.

The next time you’re considering whether or not you should automate an area of testing consider the nature of the tests. Do they use complex UIs? Do you have to repeat very similar tests over and over again? Do they currently take up a large portion of your testing cycles?

Try to automate the hidden process which run over and over and you wished you could test every time if you had the time!

Creating a Valuable Signoff Process

We all have them, a series of tests which need to be run before a build is considered good enough to be released to a customer. You may have different scripts for different stages of release (Alpha/UAT/Live), you may have different processes depending on the severity of any bugs found. Whatever your process I’m sure we can all agree that your signoff process is one of the most important pieces of your development puzzle and one of the most vital to get right in order to avoid issues further down the line.

But what makes a signoff process good? These are the things I believe a good signoff process needs to achieve:

  • Detect all unknown bugs in all critical features
  • Quick and easy to complete

Clearly these are at odds, if you want to find every bug then you’ll need to invest significant time! Let’s look at why each of these are so important before we try to find a solution.

The first one is fairly obvious, the most critical parts of your system are the ones which will generate the most urgent Support Tickets. In order to minimise those stressful late nights we need to validate that those areas of the system are as robust as possible.

But why the speed? Why not simply run through every test and permutation before each release (assuming you don’t mind driving your QA team mad)? In this world of Agile Development and quick turnarounds it’s becoming more important than ever to test and release quickly. After all, I mentioned in my previous post about Sprint Planning I suggested that you should aim to both develop and test your new features in each Sprint. In order to maximise your development potential you need to make your signoff process as efficient as possible.

So how do we do this? The key is to target your testing effectively. You need to work with your Product Owner to identify the Critical Functionality which must not fail, these form the basis of your signoff scripts. Other areas of the system can be tested on a rotational basis or when changes are made.

This prioritisation of the most critical functionality guarantees that the vital happy paths are always tested. This leaves more time for the QA team to expand their efforts into other areas while the developers are coding new features. By targeting your signoff scripts you can guarantee a high quality build without the lengthy delays which come from a bloated signoff process.

The Importance of Testing Early

I recently had a conversation with a Development Manager at a company based in Leeds. We were discussing when to involve the QA Team in a release we were planning, I argued that there was little value in wasting the QA guys’ time until we were feature complete. After all, everything was still subject to change and they’d only have to repeat those test again at a later date.

Ironically I now hold the opposite view.

If you walked up to me today and asked at what stage of development you should bring QA resource into a project I would always advise that as soon as the developers start coding it’s too late.

Your QAs are not automated test machines, I can crank out a few Selenium scripts to test a UI during my lunch hour! Your QA team are there to ensure that the features you deliver are the highest quality they possibly can be. So when does quality begin? I would argue in the design phase!

I’m currently working with a QA who, for a variety of reasons is trying to work out all a feature’s permutations eighteen months after the design was originally done. He’s documenting these, generating Functional Tests for them, and raising bugs where required. This is incredibly time consuming and takes lots of time from him, a development resource, and the Product Owner. Imagine if he’d had the opportunity to work this out before development work had begun!

The key here is to allow you Product Owner, QA, and Developer to create the spec together. The developer sets to work and the QA begins creating their functional tests, as soon as the feature is code complete your QAs are ready to go!

So, my original concern was that our testers would have to continue to test over and over again. Yes, this is a risk, however, when would you rather be alerted to any issues… as the developer is adding finishing touches, lining up buttons and tidying Unit Tests, or six weeks after they’ve finished? I know which I’d prefer!

This is where the distinction between Functional Tests and Signoff Tests becomes important. Functional Tests are used to test every permutation of a feature, to verify it against the spec, and to perform regression testing after substantial change. Signoff Scripts are to protect your critical functionality. Use your Functional Tests early to ensure that the newly created feature behaves according to spec, use your Signoff Scripts to verify your functionality before a release.

Get your QAs involved in your spec documents, organise your Sprint so they create tests while the developer codes, and get timely feedback on your features while you’re still in a position to fix them.

Multiple Binding Attributes in SpecFlow

I recently discovered something rather nice in SpecFlow, I was implementing a scenario like this

Scenario: Save a user
Given I have a user with the name Joe Bloggs
When I save them
Then the user should have a FirstName Joe
And the user should have a LastName of Bloggs

I wanted to provide the flexibility in the assertions so our QA could decide how he wanted to phrase the text in the scenario. Logically however we’d want the same binding for each variation.

Here’s what I came up with:

[Then(@"the user should have a (.*) (.*)")]
[Then(@"the user should have a (.*) of (.*)")]
public void ThenTheUserShouldHaveA(string field, string value)
  var user = GetUser();
  Assert.AreEqual(user.Properties[field], value);

However this didn’t work, I kept getting a field of “FirstName of”. I discovered however that you can reverse the binding attributes to give priority.

Updating the attributes to

[Then(@"the user should have a (.*) of (.*)")]
[Then(@"the user should have a (.*) (.*)")]

This change gave the of binding precedence and ensured that both scenario steps worked correctly.

SpecFlow what is it and why should I care?

I was introduced to SpecFlow a year or so ago by a friend of mine during a presentation at Agile Yorkshire. He described it as a tool for BDD and I was immediately struck by the revolutionary concept of human readable tests. It’s taken me a little time, some projects and many follow up questions but I feel I’m really beginning to grasp the power of this easy to use tool.

So what is it? SpecFlow is a framework which translates text into a series of steps which are compiled as a test.

This makes more sense with an example. Let’s say I’m the developing the software for a cash machine.

Given I have £150 in the bank
When I withdraw £100
Then I should have £50 remaining in my account

SpecFlow will then generate three steps for you to implement.

public class Bindings
  [Given(@"I have £(.*) in the bank")]
  public void GivenIHaveInTheBank(int p0)

  [When(@"I withdraw £(.*)")]
  public void WhenIWithdraw(int p0)

  [Then(@"I should have £(.*) remaining in my account")]

  public void ThenIShouldHaveRemainingInMyAccount(int p0)

I’ve filled these in quickly to give you an idea of how the process is intended to work.

public class Bindings
  private readonly CashMachine _cashMachine;

  public Bindings(CashMachine cashMachine)
    _cashMachine = cashMachine;

  [Given(@"I have £(.*) in the bank")]
  public void GivenIHaveInTheBank(decimal initialBalance)
    _cashMachine.Balance = initialBalance;

  [When(@"I withdraw £(.*)")]
  public void WhenIWithdraw(decimal withdrawlAmount)



  [Then(@"I should have £(.*) remaining in my account")]
  public void ThenIShouldHaveRemainingInMyAccount(decimal expectedRemainingBalance)
    Assert.AreEqual(expectedRemainingBalance, _cashMachine.Balance);

Behind the scenes SpecFlow has built a test using your test framework of choice (MSTest, NUnit, XUnit or a variety of others). These tests can be run within Visual Studio as any other unit test. Here’s my scenario running in Resharper:


Resharper Executing Test


I think most people would be impressed by this. SpecFlow has converted human readable text and enabled us to write a simple unit test for it. We can easily add a second to make sure that our new state of the art cash machine can also dispense pennies.

Given I have £100.86 in the bank
When I withdraw £1.99
Then I should have £98.87 remaining in my account

We can add additional complexity such as overdrafts

Given I have £100 in the bank
And I have an overdraft of £50
When I withdraw £125
Then I should have £-25 remaining in my account

Or functionality to check that I cannot go overdrawn

Given I have £100 in the bank
And I have an overdraft of £0
When I withdraw £125
Then I should be prevented from withdrawing money
And I should have £100 remaining in my account

I’m sure you can see the value of having simple, understandable definitions of what a particular area of code should do instead of the complicated, developer-only Unit Tests which litter many of today’s solutions. If you wish these executable tests can form the basis of your specs, documentation and even training. Your Product Owner can get involved and validate the tests are correct, if they’re particularly tech-savvy they may even write a few themselves!

What I particularly love about SpecFlow however is the way it empowers today’s QAs. From the moment the first bindings have been created your QA team can be let loose to create new tests, verify edge cases and prove alternate scenarios until they are happy that the feature works as designed. Tests can now be created alongside the features being developed, Acceptance Tests can be created directly from the specification which the QA team can use to validate each feature and scenario as it is being developed. These same tests then form the basis of your regression suite for years to come.

Should any of these tests fail the QA team can provide the developer with not only the human readable replication steps but with with an executable test which can be debugged on the dev’s own PC.

I find this very exciting, we’ve started using SpecFlow for a number of projects and I have every intention of using it for more. If you’re interested in finding out more visit the SpecFlow website or read their Quick Start Guide.