This article is an excerpt from one of the chapters of my upcoming book, End-to-End Testing with TestCafe. If you're interested in learning more about the book, visit https://testingwithtestcafe.com, or enter your email address below to receive the first three chapters of the book free and a discount not available anywhere else.

Want to boost your automation testing skills?

Enter your email address below to receive the first three chapters of the End-to-End Testing with TestCafe book for free.

End-to-End Testing with TestCafe Book Cover

When discussing TestCafe's Selector API in Chapter 8, we covered that fetching data using a selector's properties is asynchronous. When you use a property on a selector like innerText or clientWidth to get data from a targeted element, TestCafe returns a JavaScript Promise. If you want to retrieve the data for the selector's current state, you would need to resolve the promise, which you can do using the await keyword.

When running an assertion in a TestCafe test, you can use a promise from a selector or a client function to get a value in the application under test. However, using one of these promises in an assertion doesn't require you to specify the await keyword to get the necessary data for the test to perform its validation. Why is this the case? It's because of how TestCafe deals with these promises in assertions, which they call the Smart Assertion Query Mechanism.

Let's briefly touch upon a typical testing scenario to explain what the Smart Assertion Query Mechanism does. Many testing methodologies and tools perform actions synchronously. The test performs an action, the action occurs to completion, and you can execute an assertion immediately.

Assertions can be performed immediately after a synchronous action.

Modern web applications perform actions asynchronously. You can't immediately execute an assertion after performing asynchronous operations because you don't know whether the action has finished what it needs to do. Factors like network connectivity, server performance, and the system executing the tests make the time to complete an action unpredictable. You have no guarantees about how long it will take.

In other testing frameworks, you can't immediately execute an assertion on asynchronous actions.

In other testing frameworks, testers often deal with these problems by explicitly setting waits and different timeouts before executing an assertion. Doing this gives the application under tests sufficient time to finish performing any actions before running the assertion. These adjustments provide the tests with some breathing room to allow an action to complete its work.

A common solution to assert after asynchronous actions is by explicitly waiting to run the assertion.

However, using explicit waits and increasing timeouts during tests has significant issues. These strategies are unstable in end-to-end tests because you won't have complete control over the environment that runs the web application. You won't know whether your actions will perform in an appropriate amount of time, leading to inconsistent results. For example, your tests can suddenly fail if the server hosting the application is under heavy load and doesn't respond fast enough. These failures aren't useful because it's not an issue with your app. It's caused by external factors that you can't reliably predict in any test environment.

Even with explicit waits, there's no guarantee the asynchronous action will finish on time.

When tests fail due to these kinds of inconsistencies, one quick-fix solution many testers reach for is increasing the amount of time to wait before performing the assertion. It may temporarily fix the issue, but it won't guarantee that the tests won't ever fail again.

Also, setting long waits and timeouts before performing assertions will slow down your test suite significantly. Adding a few extra seconds of wait time here and there eventually adds up to minutes of wasted time. The higher the amount of time waiting, the longer your test suite takes to finish. The feedback loop between development and testing can increase to the point where the test suite is useless and actively hindering the team's progress.

Increasing waits slows down test execution significantly.

To combat this common issue when performing validations that involve asynchronous functionality, TestCafe intelligently executes assertions multiple times within a predetermined timeout period. Instead of running a validation once and failing when it doesn't see the data for comparison, it repeatedly requests the data from the promise for the assertion. If the assertion fails during the specified timeout period, TestCafe tries again and again until the assertion passes.

TestCafe attempts an assertion multiple times when performing an asynchronous action.

If the timeout period finished and the assertion was never successful during that time, TestCafe stops this process and marks the assertion as failed.

If the assertion period times out, TestCafe will stop executing an assertion.

All of your TestCafe tests already use this mechanism without you having to write any additional code or execute your test suite with a specific command. An excellent example of how the Smart Assertion Query Mechanism works is the test case you wrote earlier covering the functionality to post a new message in TeamYap's Feed section:

test("Logged-in user can create new feed post", async t => {
  await feedPageModel.createNewPost("Welcome to TeamYap!");

  await t
    .expect(feedPageModel.firstFeedPost.innerText)
    .contains("Welcome to TeamYap!");
});

The test goes through creating a new post in your TeamYap space and validating that the action works as expected. When a TeamYap user enters a message and clicks on the button to submit the form, it sends a request to the server. The server takes the provided data and processes it, storing it in a database if it's valid. When the server finishes its processing, it sends the response back to the browser, indicating whether the action succeeded or not.

The browser responds appropriately by adding the new message to the view or displaying an error message if something went wrong. This action happens asynchronously, with the browser never reloading the page during the entire process.

In other testing frameworks that don't consider asynchronous functionality for web applications, you would need to explicitly tell the test to wait a few seconds to ensure the server processed the data and returned the expected response:

test("Logged-in user can create new feed post", async t => {
  await feedPageModel.createNewPost("Welcome to TeamYap!");

  /*
    Other testing frameworks need to set an explicit wait
    here before performing the the assertion, to give the
    above asynchronous action enough time to finish.
  */
    
  await t
    .expect(feedPageModel.firstFeedPost.innerText)
    .contains("Welcome to TeamYap!");
});

Adding it to one test is not a problem, but you will often need to add these types of explicit waits in most - if not all - of your tests. One of the most significant pain points with end-to-end testing frameworks that need these timeouts is having these waits littering the codebase, making the tests challenging to follow and maintain.

Under normal circumstances, the time spent to post a new message on TeamYap takes less than one second. In other testing frameworks, you may set a 3-second timeout to give the assertion a buffer in case the action takes over a second. But what if the test server's network temporarily gets slammed with traffic while you're running the test, or the server starts running a background task that puts the system under heavy load? The response may take well over three seconds, leading to your test failing occasionally.

TestCafe is not immune to these issues. But TestCafe handles the problem more gracefully by allowing you to increase the Smart Assertion Query Mechanism timeout per assertion (using the timeout option) or for your entire test suite via a global configuration setting. Also, TestCafe has a setting to retry failing tests called quarantine mode for isolating inconsistent tests. We'll discuss these options later in this book.

The Smart Assertion Query Mechanism is one of TestCafe's most useful features since it allows you to test asynchronous functionality without having to deal with potential issues like slow network connectivity or temporarily unresponsive systems. This mechanism is instrumental for increasing the stability of your end-to-end tests, reducing the risk of errors caused by flaky tests.

If you found this article useful, you can pre-order the End-to-End Testing with TestCafe book at https://testingwithtestcafe.com and receive $10 off the book's original price when pre-ordering before the expected release date (on or before July 15, 2020).

You can also enter your email address below to receive the first three chapters of the book for free. In addition to the book sample, you'll get an exclusive discount to buy the book when it's released.

Want to boost your automation testing skills?

Enter your email address below to receive the first three chapters of the End-to-End Testing with TestCafe book for free.

End-to-End Testing with TestCafe Book Cover