When starting to work with testing (both unit testing and UI testing), one thing that every single developer starts encountering at some point is flakiness.

Flakiness is what happens when tests don't run consistently - when you get different outcomes depending on time, which machine they're being run on, or whether they're run on CI or on the computer you work on. It's one of those things that makes people say "It works on my machine", and can lead to a lot of frustration. In order to be able to use testing in a good way, you need to be able to trust your tests, and not have them slow your work down.

This week, let's take a look at some easy-to-apply tips and tricks that can help you reduce flakiness in your tests, and make them more predictable and easier to run in any environment.

Making the right assumptions

Writing tests is all about making assumptions - and then verifying that they are true. For example, when testing a simple add(numberA:numberB:) function, we make the assumption that the outcome will be the sum of numberA and numberB, and then we assert that our assumption is true, like this:

func testAddingNumbers() {
    let sum = add(numberA: 5, numberB: 3)
    XCTAssertEqual(sum, 8)
}

But sometimes we go a bit too far in making assumptions, and this is usually what leads to flakiness. For example, let's say that we are saving a document in our app to disk every 5 seconds. To verify that it works, we might write a test like this:

func testSavingDocument() {
    let document = Document()
    documentController.open(document)

    // An expectation is used to make the test runner wait for an
    // async operation to be completed.
    let delayExpectation = expectation(description: "Waiting for document to be saved")

    // Fulfill the expectation after 5 seconds
    // (which matches our document save interval)
    DispatchQueue.main.asyncAfter(deadline: .now() + 5) {
        delayExpectation.fulfill()
    }

    // Wait for the expectation to be fulfilled, if it takes more than
    // 5 seconds, throw an error
    waitForExpectations(timeout: 5)
    XCTAssertTrue(document.isSaved)
}

While the above test may work perfectly fine when running it in Xcode on your machine, there's a lot of room for flakiness here. The problem is that we assume that saving a document will always take exactly 5 seconds, so a slight delay (caused by, for example, disk access delays on CI due to resources being shared with many other projects) will make the test fail. This will soon lead to inconsistent test runs.

Removing waiting time

Instead, let's apply a similar technique as the one used in "Time traveling in Swift unit tests", and create a DocumentSaver protocol that we can mock in our test:

protocol DocumentSaver {
    func save(document: Document, after delay: TimeInterval)
}

class DocumentSaverMock: DocumentSaver {
    private(set) var document: Document?
    private(set) var delay: TimeInterval?

    func save(document: Document, after delay: TimeInterval) {
        // Capture the input so that we can use it for assertions in our test
        self.document = document
        self.delay = delay
    }
}

We can now write a test completely without using waiting times, expectations or delays 🎉:

func testSavingDocument() {
    let saver = DocumentSaverMock()
    documentController.saver = saver

    let document = Document()
    documentController.open(document)

    // Verify that the delay is indeed 5 seconds, and that the
    // correct Document instance was passed (using ===)
    XCTAssertEqual(saver.delay, 5)
    XCTAssertTrue(saver.document === document)
}

The big advantage of doing something like the above, and being able to remove waiting times, is that your tests will become both faster and more predictable. There's way less margin for error, since you are running them in a more controlled environment, without relying on external factors like hardware timing and disk access. It also makes the test more of a "true unit test", since it only verifies that the document controller asks the saver to save.

Jumping queues

Another situation when it's very common to run into flakiness is when doing concurrent programming and jumping between different DispatchQueues. For example, let's say that we have a view controller that calls an implementation of an ImageLoader protocol in order to download an image for an image view over the network. Once done, it makes sure to update the image view on the main thread (since that is a UIKit requirement), like this:

class ImageViewController: UIViewController {
    func loadImage() {
        imageLoader.loadImage(from: viewModel.imageURL) { [weak self] image in
            DispatchQueue.main.async {
                self?.imageView.image = image
            }
        }    
    }
}

Now, even when mocking ImageLoader to not fetch over the network, we're still going to have trouble testing the above code, since it always runs asynchronously (and tests are - by default - run completely synchronously).

For example, this test will fail:

func testLoadingImage() {
    let image = UIImage()
    let imageLoader = ImageLoaderMock(image: image)
    let viewController = ImageViewController(imageLoader: imageLoader)
    viewController.loadImage()
    XCTAssertEqual(viewController.imageView.image, image)
}

The reason it fails is that by the time we reach our assert, the code that updates the image view's image has not yet been run - since it's being performed asynchronously.

In situations like this it may again be tempting to start adding waiting times and expectations, but as we've already taken a look at - that is a recipe for flakiness. Thankfully, there's an alternative technique that we can use that enables us to always perform UI updates on the main thread, while still being able to keep our tests stable and predictable.

If we think about it, what we really want to do is to avoid performing updates on background threads - meaning that if we are already on the main thread, there's no need to do an async dispatch. Let's add a small function that does just that:

func performUIUpdate(using closure: @escaping () -> Void) {
    // If we are already on the main thread, execute the closure directly
    if Thread.isMainThread {
        closure()
    } else {
        DispatchQueue.main.async(execute: closure)
    }
}

And let's use it in ImageViewController:

class ImageViewController: UIViewController {
    func loadImage() {
        imageLoader.loadImage(from: viewModel.imageURL) { [weak self] image in
            performUIUpdate {
                self?.imageView.image = image
            }
        }    
    }
}

Our test will now succeed, and we have added a test against async code without any potential sources of flakiness! 🎉

Waiting for elements to appear

The final source of flakiness that I want to give you a tip on how to avoid in this post - is when waiting for an element to appear on the screen in a UI test (If you want an introduction to UI testing, check out "Getting started with Xcode UI testing in Swift").

Let's take a look at a UI test in which we are logging into our app, and then verifying that the user's profile is shown:

func testLoggingIn() {
    let app = XCUIApplication()

    // Fill in login credentials
    let emailTextField = app.textFields["Email"]
    emailTextField.tap()
    emailTextField.typeText("[email protected]")

    let passwordTextField = app.secureTextFields["Password"]
    passwordTextField.tap()
    passwordTextField.typeText("password")

    // Tap the login button
    app.buttons["Login"].tap()

    // Make sure the user's profile is shown
    XCTAssertTrue(app.staticTexts["Your profile"].exists)
}

Most apps these days use custom animations and transitions, and let's say that we are doing such a transition between the login screen and the profile screen (maybe a classic 3D flip, or something more fancy).

Normally, the Xcode UI test runner tries to compensate for animations and interaction delays by adding its own waiting time internally (you can see this in the test log when it says Waiting for app to idle), but custom transitions can really throw it off. The result is - you guessed it 😉 - flakiness, when a test sometimes is run through too fast, with the UI lagging behind.

Because of that we might start to see random failures from our test above, which can at first be quite tricky to debug and figure out. Thankfully, the solution is quite simple. All we have to do is to tell the test runner explicitly to wait for our transition to complete, by telling it to wait for a certain element to appear on the screen - in this case we'll use the profile text that we're expecting.

Let's write a small extension on XCTestCase that lets us do that:

extension XCTestCase {
    func wait(forElement element: XCUIElement, timeout: TimeInterval) {
        let predicate = NSPredicate(format: "exists == 1")

        // This will make the test runner continously evalulate the
        // predicate, and wait until it matches.
        expectation(for: predicate, evaluatedWith: element) 
        waitForExpectations(timeout: timeout)
    }
}

Now, we can replace our assert at the bottom of our test with the following:

wait(forElement: app.staticTexts["Your profile"], timeout: 5)

We use a really generous timeout, since UI tests can sometimes be really slow to execute on CI and older machines. The exact timeout is not really important for our logic here anyway, so it doesn't hurt to add a couple of extra seconds to avoid flakiness.

Conclusion

In this post we've taken a look at how to avoid 3 common sources of flakiness; Timing, async code, and UI testing delays.

There are of course (😢) a lot more sources of flakiness out there that this post didn't cover - and I'm sure we'll return to this topic in an upcoming post. However, the key thing to always keep in mind in order to write stable unit tests - is to make the right assumptions.

Try to avoid assuming that an operation will take exactly a certain amount of time, or that the UI will always be super responsive when running a test. And in case you really need to make some of these assumptions, make sure to document them - so that a failing test will be easier to debug - your teammates (or your future self!) will thank you 🙂

What are your favorite techniques & tips in order to avoid flakiness in tests? Let me know, along with any questions, comments or feedback you might have - on Twitter @johnsundell.

Thanks for reading 🚀