Most unit tests written using Apple’s built-in XCTest framework tend to follow a pattern made up of three steps: first, we set up the values and objects that we wish to test, we then perform a set of actions that trigger the functionality that we’re looking to verify, and finally we assert that the correct outcome was produced.

That may seem like a very simple structure, at least on the surface level, but just like when building any other kind of software, there’s a nearly infinite number of ways that we can approach each of those three steps when writing tests.

This week, let’s take a closer look at the last of those steps — asserting that our code produces the right outcome — by both exploring the XCTAssert family of functions that XCTest ships with, and also how we can construct our very own test assertions as well.

Picking the right assertion

A test assertion’s main role is to compare a certain result against a control value, and to fail the current test if those two values don’t match. The assertions that ship as part of the built-in XCTest framework all have the prefix XCTAssert, the most basic of which simply compares any boolean value against true:

// Without a custom message:
XCTAssert(aBoolValue)

// With a custom message that will be displayed in case of a failure:
XCTAssert(aBoolValue, "An unexpected result was encountered")

However, when it comes to most standard forms of verification, we rarely have to call XCTAssert directly — instead, there are a number of more specific versions of that function that we can use for different kinds of values.

For example, let’s say that we want to verify that a ContentCollection type correctly returns the right Item when queried using a specific item ID. To perform our assertion in this case, we’ll use XCTAssertEqual, which lets us verify that the returned item is equal to the one we were expecting — like this:

class ContentCollectionTests: XCTestCase {
    func testQueryingItemByID() {
        var collection = ContentCollection()

        let item = Item(id: "an-item", title: "A title")
        collection.add(item)

        let retrievedItem = collection.item(withID: "an-item")
        XCTAssertEqual(retrievedItem, item)
    }
}

While XCTAssertEqual is great in situations when we want to verify our code’s outcome against a very specific value, sometimes we might need keep our verification slightly less granular. For example, below we’re verifying that a Cache assigns an expiration date to an item when inserted — and we’re not really interested in verifying the exact date (since that’s an implementation detail of our cache itself), but rather just that any date was assigned, which can be done using XCTAssertNil and XCTAssertNotNil:

class CacheTests: XCTestCase {
    func testExpirationDateAssignedToInsertedItem() {
        let cache = Cache()
        let item = Item(id: "an-item", title: "A title")

        // It's often a good idea to verify that our initial state
        // is correct as well, especially when dealing with code
        // that includes some form of persistence:
        XCTAssertNil(cache.expirationDate(for: item))

        cache.insert(item)
        XCTAssertNotNil(cache.expirationDate(for: item))
    }
}

When picking which assertion function to use within each situation, it’s arguably just as important to consider how each potential option will behave when failing as it is to match it against the type of input that we’ll be passing into it.

As an example, let’s say that we wanted to verify that a NoteManager class successfully removed all of its notes when asked to do so. We’ll do that by verifying that our manager’s allNotes array is empty — which might make XCTAssertTrue seem like a perfect fit, as we can simply pass allNotes.isEmpty into it:

class NoteManagerTests: XCTestCase {
    func testRemovingMultipleNotes() {
        let manager = NoteManager()

        let notes = (0..<3).map { Note(id: "\($0)") }
        notes.forEach(manager.add)
        XCTAssertEqual(manager.allNotes.count, 3)

        manager.remove(notes)
        XCTAssertTrue(manager.allNotes.isEmpty)
    }
}

While the above will definitely work, the error message that we’ll get in case of a failure will simply be “XCTAssertTrue failed”, which doesn’t give us much of an indication as to what actually went wrong. If we instead could see the exact Note values that incorrectly remained within our NoteManager after our remove() call, that’d most likely make it much easier to debug such a failure.

To make that happen, let’s instead use XCTAssertEqual again, and simply compare our allNotes array with an empty one — which will show us exactly which notes that weren’t removed in case of a failure, since XCTAssertEqual always includes both of the values that it compared in any error message that it’ll generate:

class NoteManagerTests: XCTestCase {
    func testRemovingMultipleNotes() {
        ...
        manager.remove(notes)
        XCTAssertEqual(manager.allNotes, [])
    }
}

It’s of course quite difficult to predict what sort of information that we might want to get access to in order to debug future failures before they’ve even happened — but in general, the richer debugging information that we can provide our future selves when writing tests, the easier those tests tend to be to maintain and work with in the long run.

Custom assertion functions

While the default suite of XCTAssert functions definitely cover the most common use cases, there are situations in which we might want to extend that suite with a more specialized, custom assertion function that we’ve written ourselves.

An example of such a situation might be if our tests require us to verify slightly larger collections of data. Let’s say that our NoteManager from before also includes a pagination feature, which divides all added notes up into groups containing 25 elements each. To test that feature, we’re adding 50 notes to our manager, and we’re then asserting that each page contains the right subset of those added notes — like this:

class NoteManagerTests: XCTestCase {
    ...
    
    func testPagination() {
        let manager = NoteManager()

        let notes = (0..<50).map { Note(id: "\($0)") }
        notes.forEach(manager.add)
        XCTAssertEqual(manager.allNotes.count, 50)

        XCTAssertEqual(
            manager.notesOnPage(0),
            Array(notes[..<25])
        )

        XCTAssertEqual(
            manager.notesOnPage(1),
            Array(notes[25...])
        )

        XCTAssertEqual(manager.notesOnPage(2), [])
    }
}

Again, the above works perfectly fine as long as our test keeps succeeding, but if we’ll ever end up with a failure — say a single note that was misplaced — debugging that failure among an array of 25 or 50 values might become quite frustrating.

Wouldn’t it be great if we instead could see exactly which element that caused the failure, including any missing or unexpected new elements? Let’s see if we can make that happen by using Swift 5.1’s new ordered collection diffing feature. That feature will only work on iOS 13, macOS Catalina, and the other operating systems that Apple released in 2019, but that might not be an issue for our unit testing suite — even if we still ship our app itself with support for older system versions.

The ordered collection diffing API produces a CollectionDifference instance, which in turn is a collection containing Change values, that represent the changes between the two collections that are being compared. So let’s start by extending CollectionDifference with a method for converting a given change into a human-readable error message:

private extension CollectionDifference {
    func testDescription(for change: Change) -> String? {
        switch change {
        case .insert(let index, let element, let association):
            if let oldIndex = association {
                return """
                Element moved from index \(oldIndex) to \(index): \(element)
                """
            } else {
                return "Additional element at index \(index): \(element)"
            }
        case .remove(let index, let element, let association):
            // If a removal has an association, it means that
            // it's part of a move, which we're handling above.
            guard association == nil else {
                return nil
            }

            return "Missing element at index \(index): \(element)"
        }
    }
}

Using the above, we can again extend CollectionDifference with another method that lets us convert its entire set of changes into one unified error message — like this:

private extension CollectionDifference {
    func asTestErrorMessage() -> String {
        let descriptions = compactMap(testDescription)

        guard !descriptions.isEmpty else {
            return ""
        }

        return "- " + descriptions.joined(separator: "\n- ")
    }
}

Above we’re making use of the fact that Swift supports first class functions, which enables us to pass our testDescription(for:) method as if it was a closure.

Finally, let’s write our new, custom assertion function — which we’ll simply call assertEqual (we shouldn’t use the XCT prefix since we’re defining this new function outside of the XCTest framework itself). Since the ordered collection diffing API is only compatible with collections that conform to BidirectionalCollection, we’ll make that a requirement for our new function, while also requiring that all elements conform to Hashable.

Within our function, we’ll then use the standard library’s difference(from:) API to produce a CollectionDifference instance, which we’ll convert into an error message using the method we defined above. If that message isn’t empty, we cause our test to fail by passing that message along to XCTAssert:

func assertEqual<T: BidirectionalCollection>(
    _ first: T,
    _ second: T,
    file: StaticString = #file,
    line: UInt = #line
) where T.Element: Hashable {
    let diff = second.difference(from: first).inferringMoves()
    let message = diff.asTestErrorMessage()

    XCTAssert(message.isEmpty, """
    The two collections are not equal. Differences:
    \(message)
    """, file: file, line: line)
}

Note how we capture the file and line of the code location that our custom assertion function was called from, and we then pass that data along to XCTAssert. That way, Xcode will display any failures that our function will generate at the correct location, inline within our code.

With the above in place, all we now need to do is to replace our previous test implementation’s calls to XCTAssertEqual with assertEqual, and we’ll get much more granular (and actionable) error messages if we ever start encountering a failure:

class NoteManagerTests: XCTestCase {
    ...
    
    func testPagination() {
        ...
        
        assertEqual(
            manager.notesOnPage(0),
            Array(notes[..<25])
        )

        assertEqual(
            manager.notesOnPage(1),
            Array(notes[25...])
        )

        assertEqual(manager.notesOnPage(2), [])
    }
}

What’s really cool is that not only can the above function be used to verify the contents of ordered data structures like Array, but because we used the BidirectionalCollection protocol as its constraint, it can also be used with other collections as well — such as ranges, index sets, and strings.

However, when it comes to strings in particular, we’ll end up with a character-by-character comparison, which might be a bit too granular — so we might want to split each string that we’re verifying up a bit, for example into words, so that it’ll be easier to identify what the actual differences were:

let string = ...
let expectedString = ...

assertEqual(
    string.split(separator: " "),
    expectedString.split(separator: " ")
)

The above is yet another another example of the power of writing reusable extensions that are not bound to a concrete type, but rather to a standard library protocol — as it lets us get much more milage out of each utility that we write.

Conclusion

Picking the right set of assertion functions for each test that we write can have a big impact on both the clarity and semantics of our testing code, and also on the kind of information that we’ll get access to if a test starts failing.

While the standard suite of assertions that the XCTest framework ships with are most likely going to be enough to cover the needs of most code bases, also being able to create our own assertion functions when needed is incredibly powerful — and can help us build up our own “library” of testing utilities, that in turn can make it easier and easier to write new tests.

What do you think? How do you tend to perform the assertions within your unit tests, and have you ever written a custom assertion function? Let me know — along with your questions, comments and feedback — either via email or Twitter.

Thanks for reading! 🚀