When writing tests for any application, it's always important to consider what the debugging experience will be like when they eventually start failing. After all, the purpose of all kinds of automated tests is to eventually fail - since it'll let us catch bugs and errors before they reach our users. So optimizing for that use case becomes an essential part of building up a test suite that is easy to maintain and work with.

This week, let's take a look at a few different scenarios, and how we - with just a few subtle tweaks - can make our tests a lot easier to debug. Let's dive right in!

Structure

Just like how we want our production code to conform to a well-defined structure that is easy to follow - it's equally important to consider how we structure our tests. Like we took a look at in Avoiding force unwrapping in Swift unit tests, treating our test code with the same amount of care and attention to detail as our production code can make a big difference in how maintainable our tests become.

Using Given, When, Then is a popular paradigm that provides such a structure in a lightweight, simple way. It's not so much about how tests are named or how dependencies are set up, but more about the flow of the test code we write.

The idea is that we split each test method up into three parts:

• Given: Where we set up all of our dependencies and define our data.
• When: Where we perform the operations that we want to test.
• Then: Where we verify that the correct output was produced.

Let's take a look at an example, in which we're unit testing a BookmarkManager, to make sure that a bookmark can be added:

class BookmarkManagerTests: XCTestCase {
    func testAddingBookmark() {
        // Given
        let manager = BookmarkManager()
        let url = URL(fileURLWithPath: "URL")
        let bookmark = Bookmark(name: "Bookmark", url: url)

        // When
        manager.add(bookmark)

        // Then
        XCTAssertTrue(manager.contains(bookmark))
    }
}

As you can see above, applying some form of structure (whether it's Given, When, Then, or something different) can really help make our tests a lot easier to read - which becomes very helpful when tests eventually start failing and we have to figure out why.

Verifying assumptions

Our test above has a neat structure, but it actually has a couple of problems. First of all, we have made a few assumptions that might make debugging a bit harder than it needs to be.

What's not obvious is that BookmarkManager actually uses a database under the hood, which defaults to a database that persists all records (which is what we want in production). That means that our test might actually pass even though it shouldn't, since the underlying database might already contain a record for the added bookmark, even before our test code adds it.

Essentially, we have made an assumption that BookmarkManager is always empty when created. Whenever we make an assumption like that, it's usually a good idea to pair it with an assert, just to verify that our assumption is correct. In this case, we'll use XCTAssertFalse right after our Given section to assert that our BookmarkManager doesn't already contain our bookmark:

// Given
let manager = BookmarkManager()
let url = URL(fileURLWithPath: "URL")
let bookmark = Bookmark(name: "Bookmark", url: url)

XCTAssertFalse(manager.contains(bookmark))

Since we are using our production database, we'll eventually start seeing failures when adding the above, which is great - since it'll give us an entry point for debugging.

What we'll do to fix this problem is to implement an InMemoryDatabase - which is a database implementation that only stores records in memory - which we'll then use in our test:

// Given
let database = InMemoryDatabase()
let manager = BookmarkManager(database: database)
let url = URL(fileURLWithPath: "URL")
let bookmark = Bookmark(name: "Bookmark", url: url)

XCTAssertFalse(manager.contains(bookmark))
XCTAssertEqual(database.records(withType: Bookmark.self), [])

As you can see above, we've also added yet another assumption verification in that we're verifying that our database doesn't yet contain any Bookmark records. Verifying our initial state like that can really help improve the stability of our tests, and gives us hints as early as possible as to why a test is failing.

Note that InMemoryDatabase is not a mock, it's a proper database implementation that's not only useful for testing, but also for debugging - for when you always want to start the app in a clean state. By following the principles from "Separation of concerns using protocols in Swift", swapping database implementations in our tests should be easy - since BookmarkManager only relies on a protocol.

Finally, let's verify some of the assumptions that we've made in the Then section as well - by making sure that the added bookmark also has been associated with the correct URL, as well as verifying that a record has been added to the database:

// Then
XCTAssertTrue(manager.contains(bookmark))
XCTAssertEqual(manager.bookmark(withURL: url), bookmark)
XCTAssertEqual(database.records(withType: Bookmark.self), [bookmark])

With all the above in place, it will now be much easier to debug this test if it starts failing, since we'll have clear information as to what exactly failed and where we need to start looking for the underlying cause.

Reusable utilities

When writing tests, we ideally want to keep our test methods as simple and short as possible. The Given, When, Then structure will start falling apart quite quickly if we have to do a ton of setup before we can start testing, which again will make debugging much harder.

One thing that can help us achieve that goal, is to extract common utilities from our tests into helper functions and extensions. For example, let's say our app performs a lot of networking, so we constantly need to mock network responses when creating an instance of our DataLoader class in our tests. To make that easy, let's add an extension on DataLoader that lets us pass a dictionary of mocked responses with their URLs as keys:

extension DataLoader {
    static func makeMock(with responses: [URL : Response]) -> DataLoader {
        let engine = NetworkEngineMock(responses: responses)
        return DataLoader(engine: engine)
    }
}

With the above in place, we can now remove all initializations of NetworkEngineMock from our test cases, and instead simply do this whenever we want to mock a network response:

let dataLoader = try DataLoader.makeMock(with: [
    Endpoint.productList.url : [Product].makeStub().makeResponse()
])

As you can see above, we're using a similar pattern to make an array of stubbed products and then turn them into a Response struct. For more on that pattern, check out last week's article - "Static factory methods in Swift".

Very nice and clean - and since we'll use this utility all over our test suite, we can often rule it out as the cause of a problem when only a single test starts failing.

Reusable navigation in UI tests

The same practice of reusing utilities can also be applied to UI tests. For example, we might find ourselves needing to login to our app in many different tests. Instead of having to put login code inline in every single test method, let's instead create an extension on XCUIApplication (which is the class that Xcode's UI testing framework uses to represent our app), that lets us login with one line of code:

extension XCUIApplication {
    func login(withUsername username: String,
               password: String = .uiTestingPassword) {
        // Verify that we're on the login screen, or fail early
        let navigationBar = navigationBars["Login"]
        XCTAssertTrue(navigationBar.exists,
                      "The app is not on the login screen")

        let usernameTextField = textFields["Email"]
        usernameTextField.tap()
        usernameTextField.typeText(username)

        let passwordTextField = secureTextFields["Password"]
        passwordTextField.tap()
        passwordTextField.typeText(password)

        let loginButton = buttons["Login"]
        loginButton.tap()
    }
}

Important to note above, is how we start our login method by verifying that we're actually on the login screen. Again, this is verifying our assumption that this will always be called when we're able to login. Adding that extra check can make tests using this utility a lot easier to debug, since it'll clearly tell us if the app is not displaying the login screen as expected.

We can now use the same simple Given, When, Then structure for our UI tests as well, since all we're really doing is executing commands that we have pre-defined using XCUIApplication extensions:

func testOpeningFriendsList() {
    // Given
    let app = XCUIApplication()
    app.launch()
    app.login(withUsername: "ui-tester-with-friends")

    // When
    app.openFriendsList()

    // Then
    XCTAssertEqual(app.isFriendsListOpen)
}

To learn more about UI testing, check out "Getting started with Xcode UI testing in Swift", as well as my UIKonf talk "The Magic of UI Testing".

Localized errors

Finally, let's take a look at how we can make tests easier to debug by making it easier to identify any underlying errors that might be thrown. Here we're testing an HTMLParser to make sure that it's correctly parsing a given string into HTML tags:

class HTMLParserTests: XCTestCase {
    func testParsingNestedTag() throws {
        // Given
        let parser = HTMLParser()
        let string = "<html><div></html>"

        // When
        let rootTag = try parser.parse(string)

        // Then
        XCTAssertEqual(rootTag.name, "html")
        XCTAssertEqual(rootTag.children, [Tag(name: "div")])
    }
}

As you might have noticed above, this test will currently fail, since we're not closing the <div> tag when defining our HTML string. Here's what Xcode will display when that failure happens:

HTMLParserTests.testParsingNestedTag() failed: failed: caught error: The operation couldn’t be completed. (ErrorsTests.HTMLParser.Error error 0.)

Although it gives us an idea of what type of error that was thrown, it doesn't provide us with any kind of rich debug information. Thankfully, it's easy to fix. All we have to do is make our error type conform to LocalizedError, which is an extension of Swift's Error protocol, and implement errorDescription - like this:

extension HTMLParser.Error: LocalizedError {
    var errorDescription: String? {
        return "\(self)"
    }
}

If we run the same test again, we now get a nice error message that - since we're running in debug - defaults to the error's debugDescription, which in this case gives us exactly the kind of debug information that we need:

HTMLParserTests.testParsingNestedTag() failed: failed: caught error: missingClosingTag("div")

We can now much easier identify what went wrong and fix our test 🎉.

Conclusion

As a test suite grows and becomes more sophisticated, it becomes increasingly important to treat it as a proper system that we need to actively maintain. By making small tweaks to the way we setup and write our tests, we can save ourselves (and others) a lot of time in the future, once we need to start debugging why a test failed.

Writing tests in a more easily debuggable manner can also help identify flakiness as early as possible, and Xcode 10 even includes some new features - like randomizing the test execution order - that can further help us in that regard.

Another very common source of frustration and debugging difficulties is asynchronous tests. While we've already taken a look at how to test async code before - it's a topic we'll soon revisit in terms of how we can make such tests even easier to work with.

What do you think? Do you currently apply some of the practices from this article when writing tests, or will you try some of them out? Let me know - along with your questions, comments and feedback - on Twitter @johnsundell.

Thanks for reading! 🚀