I’ve lost track of the number of times I’ve walked into the office on a morning to be faced with a wall of failed SpecFlow tests. I care about my SpecFlow tests, I look after my SpecFlow tests, I feed my SpecFlow tests and I might even love them. Even so, it still doesn’t stop them failing on me for the most minor of reasons. Here’s some tricks, tips and techniques that I have used to write more robust SpecFlow tests.

What are SpecFlow tests?

SpecFlow tests are a way to write automated tests in a natural language format. This lends itself to tests focused around user stories and behaviour driven development working practices.

A test would be something like

Scenario: Retrieve a list of books
       Given I have searched for a book about 'dinosaurs'
       Then I retrieve a page of 10 results

The test should make sense to all members of a project team irrespective of their technical know-how and background. Each individual step maps to a method via a regular expression

[Given(@"I have searched for a book about '(.*)'")]
public void GivenIHaveSearchedForABookAbout(string searchTerm)
{
  var session = new Mock<ISessionKeys>();
  var bookService = new GoogleBookService();
  var bookController = new BookController(bookService, session.Object);
  _result = bookController.Search(searchTerm, 1);
}

When built each test is transformed into NUnit which can then be run by the NUnit test runner

[NUnit.Framework.TestAttribute()]
[NUnit.Framework.DescriptionAttribute("Retrieve a list of books")]
public virtual void RetrieveAListOfBooks()
{
  //.. test gubbins
}

It’s useful to bear in mind that really these tests are NUnit tests and can be treated as such. SpecFlows can be configured to transform into different test frameworks such as MSTest but NUnit is the default.

I’m assuming that the spec flow tests are running on some kind of continuous build cycle. If they aren’t then really that is the top tip – get them running on a schedule in Team Foundation Server, Team City or the like. Once you have your tests running on some kind of schedule then it’s time to toughen them up.

1. Run the most important tests more frequently

If you have plenty of tests, a useful thing to do is split them up into smaller groups. I have got a full set of SpecFlows that take well over an hour to run which are often failing due to minor disturbances. I’ve got a smaller group of tests that take 15 minutes to run that I’ve identified as being particularly critical to the running of the system. I can accept the full set being red (although I’m not hugely happy) but I really need to see the smaller group of critical test passing all the time.

Example

Feature: BookShelf
       In order manage my virtual bookshelf
       As an reader of books
       I want to add and remove books to my shelf

Background:
       Given I am using the virtual bookshelf

@important
Scenario: Retrieve a list of books
       Given I have searched for a book about 'dinosaurs'
       Then I retrieve a page of 10 results

@important
Scenario: Finding a particular book
       Given I have searched for the book 'harry potter and the chamber of secrets'
       Then 'harry potter and the chamber of secrets' is on the first page of results

Scenario: A less important test
       Given I am using the virtual book shelf
       When I am browsing
       Then I don't want to be logged out

Given the above tests I have identified the first two as being of particular importance. The last one I’m less concerned about. So I have tagged the first two as @important and I can use that to limit the run to the first two.

The @important tag is transformed into a category attribute on the underlying NUnit test i.e.

[NUnit.Framework.CategoryAttribute("important")]
public virtual void RetrieveAListOfBooks() 
{
   //.. test gubbins
}

Since these are just unit tests under the covers I can use the NUnit command line to just run the spec flows that I have deemed the most important i.e.

nunit3-console.exe BookShelf.Specflow.dll --where "cat == important”

will just run the first two and ignore the last one.

It’s far easier to keep your most important 20% of tests running than it is to keep all of them up and running. Your 20% are the baseline to your applications health. If they aren’t running, then it’s red alert.

Running continuously

This technique can be rolled into your continuous builds where it is most useful. To take the example of Team Foundation Server, the category can be specified in the build definition under TestCaseFilter.

Team city offers the same functionality.

This technique could be extended to split SpecFlow tests into functional areas so it becomes obvious that the tests are failing in a particular part of the application where other aspects of the system may be very robust and healthy. It also could be inverted to exclude the most fragile tests and keep the majority of the tests that you are most confident about in the continuous build runs.

2. Stress testing

It’s always difficult to deal with tests that fail occasionally. If you have 200 tests and each one fails 1% of the time then you will only get a full pass of your integration tests 13% of the time. If you’ve got a good set of robust tests but 20 of these are flaky and fail 10% of the time then at least one of these bad apples will fail 88% of the time. On an overnight build you will see a fully passing green build once a fortnight. This hardly inspires a massive amount of confidence in your software quality. Of course when you run these bad boy tests individually they will probably pass and claim everything is OK. Frustrating.

However it is possible to run these tests over and over and catch them in the act of failing. The key is to remember that they are really unit tests and can be treated as such.

Example

Considering this test

Scenario: Finding a particular book
       Given I have searched for the book 'harry potter and the chamber of secrets'
       Then 'harry potter and the chamber of secrets' is on the first page of results

When built it transforms into this unit test in a class named after the feature

[NUnit.Framework.TestFixtureAttribute()]
[NUnit.Framework.DescriptionAttribute("BookShelf")]
public partial class BookShelfFeature
{
  [NUnit.Framework.TestAttribute()]
  [NUnit.Framework.DescriptionAttribute("Finding a particular book")]
  public virtual void FindingAParticularBook()
  {
     //.. test gubbins
  }
}

We can then create a wrapper unit test to rerun this spec flow scenario as many times as we like

[TestFixture]
public class BookShelfStressTest
{
  private const int MAXTEST = 50;

  [Test]
  public void BookSearchStressTest()
  {
    var feature = new BookShelfFeature();
    feature.FeatureSetup();
    feature.TestInitialize();

    for (int i = 0; i < MAXTEST; i++)
    {
       Console.WriteLine("----- Attempt {0} -----", i);
       feature.FindingAParticularBook();
       feature.ScenarioCleanup();
    }
  }
}

The only thing to remember is to set up the feature beforehand and clean up afterwards. With some decent logging in the spec flow tests you’ll be able to identify why it fails 1 time in 20 and fix it. I’ve really found this technique useful in getting rid of intermittent failures.

3. Improve logging in the tests

Which brings us neatly onto writing good test logging. It’s a minor point but like a lot of minor issues it can have a really significant benefit. It is a generally useful thing to instrument your SpecFlows so you can see what is happening when they fail (again) overnight.

Example

Scenario: Retrieve a list of books
       Given I have searched for a book about 'dinosaurs'
       Then I retrieve a page of 10 results

The ‘Then’ statement is implemented by the following step.

[Then(@"I retrieve a page of (\d+) results")]
public void ThenIRetrieveAPageOfResults(int pageNumber)
{
   var searchResults = _result.Model as PagedResults<Book>;
   searchResults.Should().NotBeNull();
   searchResults.Results.Count.Should().Be(10);
}

This won’t tell us a huge amount when it fails but it can be quickly improved by

[Then(@"I retrieve a page of (\d+) results")]
public void ThenIRetrieveAPageOfResults(int pageNumber)
{
  var searchResults = _result.Model as PagedResults<Book>;

  Console.WriteLine("Search results " + JsonConvert.SerializeObject(searchResults));
  searchResults.Should().NotBeNull("Search results are null");
  searchResults.Results.Count.Should().Be(10, "Nunber of result found were {0}", searchResults.Results.Count);
}

The console command will write out to the output pane of whichever test runner I am using. I like to serialise any object to JSON so I’ve got even more detail on the output. Maybe the object has a helpful ToString method that I can use but it probably doesn’t so JSON serialisation is useful here. Also the assertions have been amended to give more detailed output when they fail. My output is much improved.

Test Name:    RetrieveAListOfBooks

Test Outcome:           Passed

Result StandardOutput:

Given I am using the virtual bookshelf

-> done: BookShelfSteps.GivenIAmUsingTheVirtualBookshelf() (0.3s)

Given I have searched for a book about ‘dinosaurs’

-> done: BookShelfSteps.GivenIHaveSearchedForABookAbout(“dinosaurs”) (1.5s)

Then I retrieve a page of 10 results

Search results {“Results”:[{“BookID”:null,”BookIDSource”:”tAr9XAv136kC”,”Title”:”Dinosaurs!”,”ISBN_10″:”0307982696″,”ISBN_13″:”9780307982698″,”Description”:”Dinosaurs! follows the evolution of these spectacular creatures from their earliest beginnings as little fellows who had to evade attacks from giant croc relatives to today’s living dinosaurs.”,”Rating”:0.0,”PageCount”:24

… truncated …

PageSize”:10,”Pages”:20}

-> done: BookShelfSteps.ThenIRetrieveAPageOfResults(10) (0.3s)

 

4. Use fuzzy string matching

I find that failed string matches often trip up the tests. Often the tests are checking validation strings or results from a search and minor variations can cause a failure. I’ve found it useful to employ fuzzy string matching to mitigate this.

Example

Considering the following test

Scenario: Finding a particular book
       Given I have searched for the book 'harry potter and the chamber of secrets'
       Then 'harry potter and the chamber of secrets' is on the first page of results

It could be that the return results don’t quite match. Perhaps the result isn’t pluralised or has a minor typo. I don’t want to declare the search function broken in those circumstances. The standard way to implement the ‘Then’ statement would be

[Then(@"'(.*)' is on the first page of results")]
public void ThenIsOnTheFirstPageOfResults(string bookTitle)
{
  var searchResults = _result.Model as PagedResults<Book>;
  searchResults.Results
            .Any(x => x.Title.ToLower() == bookTitle)
            .Should().BeTrue("{0} title not found", bookTitle);
}

However we could make this looser by installing a fuzzy matching library such as DuoVia.FuzzyStrings

The step then becomes

[Then(@"'(.*)' is on the first page of results")]
public void ThenIsOnTheFirstPageOfResults(string bookTitle)
{
  var searchResults = _result.Model as PagedResults<Book>;
  searchResults.Results
                .Any(x => x.Title.FuzzyMatch(bookTitle) >= 0.7)
                .Should().BeTrue("{0} title not found", bookTitle);
}

which will be more resilient to minor issues. Obviously you can play around with the tolerances (0.7 currently) to match your circumstances but minor differences won’t detonate your tests any longer.

5. Write fewer SpecFlow tests

I’ve saved my least helpful tip till last. If SpecFlow aren’t particularly robust then it would help to write fewer of them and more unit tests which will be more solid. I had exactly that message delivered to me by a consultant and it’s a hard message to hear. It’s made all the more harder to hear since it’s probably right.

This isn’t going to help an established project with a mountain of failing SpecFlow tests. But on a new project it might be worth trying to favour the more robust unit tests and keep the integration tests for more genuinely end to end testing. Unhelpful. Sorry.

Useful links

SpecFlow is an port of Cucumber. Project page at
http://specflow.org/

How to use the default unit test provider in SpecFlow
http://www.marcusoft.net/2010/12/appconfig-for-mstest-and-specflow.html

Details of NUnit test selection language that I used to filter the SpecFlow tests by category
https://github.com/nunit/docs/wiki/Test-Selection-Language

I like to use Fluent Assertions for my automated testing. It’s just a bit nicer syntax.
http://www.fluentassertions.com/

Fuzzy string matching and suggested tolerances can be found at
https://github.com/tylerjensen/duovia-fuzzystrings

As ever, the source code for the examples is at my GitHub site
https://github.com/timbrownls20/BookShelf