There are two problems that almost every team that uses unit testing will encounter at some point - flakiness and slowness. Flakiness is what happens when a test starts succeeding or failing based on external factors (which can at times seem random), and slowness is often a byproduct of having a large test suite which needs to run in sequence.

Xcode 10 includes two new features aimed at addressing those two common testing problems. In this year's last daily WWDC update, let's take a look at what those features are and how they can potentially make our tests faster and more robust.

Implicitly shared state

State gets implicitly shared when two pieces of code relies on the same mutable state without going through a defined, predictable API. For example, we might accidentally cause two view controllers to reference the same mutable model, and when one of those view controllers update the model, the other one ends up in an undefined state - since it still thinks its dealing with the old version of that model.

The same thing can happen when writing test code as well, and can be a common source of flakiness. Let's take a look at an example, in which we have two tests that each perform work on the same file - one test creates it, while the other one deletes it:

class FileTests: XCTestCase {
    func testCreatingAndReadingFile() throws {
        let path = "path"

        XCTAssertNil(FileManager.file(at: path))

        try FileManager.createFile(at: path, containing: "Hello")
        let file = FileManager.file(at: path)
        XCTAssertEqual(file?.read(), "Hello")
    }

    func testDeletingFile() throws {
        let path = "path"

        XCTAssertNotNil(FileManager.file(at: path))

        try FileManager.deleteFile(at: path)
        XCTAssertNil(FileManager.file(at: path))
    }
}

The above two tests will succeed when run in sequence, but as we can see when taking a closer look, we actually have some problems here. The test verifying that we can delete a file implicitly relies on the file being created by the previous test, and vice versa. That means that if we ever change the execution order, we'll start seeing flakiness and random failures.

Randomized execution order

Thankfully, Xcode 10 includes a new testing setting that can help us identify this kind of problem before it starts causing flakiness. By opening our scheme's test settings (keyboard shortcut ⌥ + ⌘ + U) and clicking the Options button, we can turn on the Randomize execution order setting. By doing that, Xcode will always randomize the order in which our tests get executed, revealing the issue with our test from above.

When running our tests again with the new setting enabled, we'll start getting failures - and we can patch our tests to instead use more predictable, local states. What we'll do in this case is always run our tests in an isolated, temporary folder - that we'll make sure to delete and (re)create before each test is run, by doing that in our test case's setUp method:

class FileTests: XCTestCase {
    let folderPath = NSTemporaryDirectory() + "FileTests"

    override func setUp() {
        super.setUp()
        try? FileManager.deleteFolder(at: folderPath)
        try! FileManager.createFolder(at: folderPath)
    }
}

With the above in place, let's update our tests to run against our new temporary folder, which'll make them always start from a clean state:

func testCreatingAndReadingFile() throws {
    let path = folderPath + "/createAndRead"

    XCTAssertNil(FileManager.file(at: path))

    try FileManager.createFile(at: path, containing: "Hello")
    let file = FileManager.file(at: path)
    XCTAssertEqual(file?.read(), "Hello")
}

func testDeletingFile() throws {
    let path = folderPath + "/deleting"

    try FileManager.createFile(at: path)
    XCTAssertNotNil(FileManager.file(at: path))

    try FileManager.deleteFile(at: path)
    XCTAssertNil(FileManager.file(at: path))
}

Our tests are now consistently passing, even when the order is randomized! 🎉 Another way to achieve the same results would be to use mocking to prevent our code from using the actual file system. For more information about using mocking in Swift unit tests, check out "Mocking in Swift".

Parallelizing

Now that we have made sure that our test suite can consistently pass, even when the execution order is randomized, we can start taking advantage of yet another new testing feature in Xcode 10 - parallel testing.

When the Execute in parallel option is enabled for a testing bundle (which can be done by editing our scheme's testing settings, just like when enabling randomization), Xcode will spawn multiple clones of the same iOS simulator or Mac app, and spread out our test cases among them. To take full advantage of this new feature, we should make sure that we don't gather too many tests in one massive test case class - since that class won't be parallelized.

For example, we might have a huge class containing a large number of tests verifying all of our database functionality:

class DatabaseTests: XCTestCase {
    func testCreatingRecord() { ... }
    func testDeletingRecord() { ... }
    func testAddingChildRecord() { ... }
    func testRemovingChildRecord() { ... }
    func testUpdatingMultipleRecords() { ... }
    ...
}

To enable parallelization, let's instead split that massive class up into multiple smaller ones, that each contain a given category of database tests:

class DatabaseCreationTests: XCTestCase { ... }
class DatabaseDeletionTests: XCTestCase { ... }
class DatabaseUpdateTests: XCTestCase { ... }

And with that, we can now take full advantage of parallelization, and our tests will most likely execute a lot faster! 👍

Doing the above can also really help when it comes to making our tests easier to work with, as we don't have to navigate one massive class containing a huge number of tests.

Conclusion

I'm really happy to see Apple continue investing in testing infrastructure, enabling us third party developers to much easier scale up our test suites and identify common sources of flakiness. As the infrastructure improves, it'll also make it much easier for all kinds of teams (not only big ones with the ability to build custom tooling) to start using testing in their day-to-day work, which will help us all build better and more stable apps.

For more information about testing, make sure to check out the Category page, that contains over 10 articles about testing in Swift. If your team doesn't yet use unit testing but want to start, you can also hire me to run a testing workshop to help you get started.

For a complete overview of Xcode 10's new testing capabilities, make sure to check out the "What's new in testing" WWDC session.

What do you think? Are you excited about these new testing improvements, and will you try using them in your app? Let me know - along with your questions, comments and feedback - on Twitter @johnsundell.

Thanks for reading! 🚀