End-to-end testing is one of the most effective ways to ensure that a web application will work for your customers as expected. Given the complexity of modern web applications, coupled with the increasing demands to build and deploy quickly, it's more important than ever to have an automated test suite handling these tests for you.
It's easier than ever to create end-to-end tests for any web application. You can find dozens of tools, libraries, and frameworks that cover most of what you need. My personal preference for these kinds of tests is TestCafe. TestCafe is a free, open-source library with tons of functionality out of the box, and it uses JavaScript so most web developers can contribute to building the test suite. You can have a fully functional, robust test suite covering the most critical parts of your web app in little time.
While TestCafe can handle most end-to-end tests in an easy yet powerful way, you'll still need to run your test suite frequently. An integral part of any automated testing pipeline is running them constantly and consistently. You want to ensure that any changes made to an application don't contain any regressions that your existing test suite can detect. The best way of handling that is by using a continuous integration service.
Running end-to-end tests for web applications on a continuous integration system can get tricky. These tests will need to run your test scenarios in a browser instance, usually on a headless environment without a graphical user interface. Also, you'd ideally want to test on different types of browsers since your web app may behave differently. TestCafe can handle all of these scenarios without the need for additional setup and configuration, making it an ideal tool for ensuring your app works on different systems.
While services such as BrowserStack and LambdaTest allow you to run your tests on multiple browsers without dealing with hardware or software requirements, this series will focus on running your end-to-end TestCafe tests within a single CI service. We'll attempt to demonstrate how to set up multiple browsers using some of the most popular continuous integration services on the market.
Other articles available in this series:
Dennis Martinez
Example test suite used in this article
For this series, we'll use a fully functional TestCafe test suite for the example application used in my book, End-to-End Testing with TestCafe. The test suite covers different scenarios to teach testers how to use TestCafe in a real-world web application called TeamYap. While the tests require access to the TeamYap application - only available when purchasing the book - you can still access the code for the entire test suite on GitHub.
dennmartCircleCI: Easy yet powerful continuous integration and delivery
The first article in this series will cover CircleCI. CircleCI is a fully-fledged continuous integration and continuous deployment service that allows developers and testers to automate the complete software development lifecycle for their projects. The company offers a cloud-based service, along with an on-premise self-hosted offering for organizations requiring enterprise-level options.
CircleCI also has multiple options for running your jobs. You can execute your builds on Linux, Windows, and macOS systems. You can also choose ARM-based systems if you need to target that architecture for your project. For specialized computation jobs, like AI, gaming, or math-heavy workloads, you can also run your jobs using GPU resources. In short, they provide options for almost any modern project build.
The primary benefit of choosing CircleCI, in my experience, is that it's easy to get up and running while containing tons of powerful features under the hood when you need them. It's also dead-simple to scale as your projects and organization grow, letting you start with a single job to multiple concurrent builds at the click of a button. It also has a generous free tier for getting started with them, as well as supporting open source projects by offering additional use time for their service.
Free tier usage
CircleCI's pricing scheme can be a bit confusing. The service charges a credit-based system, calculated on a per-minute basis depending on the resources used to run your CI jobs. For instance, running your job on their smallest Docker instance with one virtual CPU and 2 GB of memory will use five credits per minute of usage. In contrast, their medium Docker instance with double the power (two virtual CPUs, 4 GB of memory) consumes ten credits per minute.
CircleCI offers 2,500 credits per week on their free tier, which is good for up to 500 minutes on the smallest Docker instance. However, if you use CircleCI on an open-source project, you can use 400,000 credits per month as long as your code repository remains public. You're limited to running one job at a time and can only use their smaller, limited resources. Also, you won't have access to their MacOS and GPU-specific resources.
Paid tier usage
CircleCI's paid tiers are even more confusing. The paid service currently begins at $30 per month. Their paid tier gives you 25,000 credits per month for up to three users committing code to your project that runs on the service and adds 25,000 credits per month for each additional user. You can scale the number of credits as needed.
Besides more credits, the paid tier also gives you many additional options not available on the free/open-source level. You'll have access to almost all of the different instances for running your jobs, including macOS instances. You can run jobs concurrently, up to 80 at a time. It also has other niceties like priority customer support and options like Docker layer caching, which can speed up your jobs if you're using CircleCI to build Docker images for your project.
For enterprise-level users or organizations needing even more options, you can pay extra to self-host CircleCI in your data centers - necessary for projects with strict regulations and compliance. It also provides access to GPU instances on demand and other functionality geared to large companies.
Getting CircleCI up and running with TestCafe
To sign up for CircleCI's cloud service and use the free tier, you'll need a GitHub or Bitbucket account. CircleCI currently only supports these two services for signing up and authenticating users. If you use another code repository service, you won't be able to sign up with it, nor through a standard email and password setup.
Once you authenticate through GitHub or Bitbucket and grant the necessary repository permissions to CircleCI, you will see all of your existing projects for that service. Find the project containing your TestCafe test suite, and click on "Set Up Project". CircleCI will take you to a screen where you can choose a programming language to get a pre-populated configuration file to place in your project.
CircleCI works by connecting to your code repository and looking for a configuration file placed inside called .circleci/config.yml. This configuration file contains all the steps needed to build and execute your continuous integration workflow. You can choose a pre-populated configuration based on your desired programming language (like JavaScript, for instance) and commit it to your repository directly from CircleCI. However, we'll start a configuration file from scratch to learn how each piece works.
Let's begin by setting up the basics of our CircleCI configuration and explaining while we go. Inside of our project, we'll create a new directory called .circleci, and inside create a file called config.yml. This file includes the very basics that will tell CircleCI how we want to set up our builds:
version: 2.1
orbs:
node: circleci/[email protected]
browser-tools: circleci/[email protected]The first setting in our configuration, version, tells CircleCI which version of their platform we want to run our builds. All CircleCI configuration files require this setting. The latest available version at this time is version 2.1, so we'll use it for this article.
The second setting, orbs is a recent addition to CircleCI version 2.1. CircleCI orbs are pre-packaged configuration settings that allow us to reuse standard functionality across different scenarios. The CircleCI team provides orbs that handle what most projects need, but anyone can create and share their orbs to benefit other CircleCI users. For our tests, we'll use two orbs to help set up our test environment.
The first orb, circleci/node, provides functionality to set up Node.js and package managers like npm and Yarn in our test environments. The orb also contains other commands for installing dependencies and includes helpful functionality like caching installed dependencies between builds to save time in subsequent test runs. Since TestCafe is a Node.js package, we'll need this orb to get it installed along with its dependencies.
The second orb used, circleci/browser-tools, helps us install and set up browsers and associated tools for browser testing. The orb can set up Google Chrome and Mozilla Firefox, along with utilities like ChromeDriver and GeckoDriver if you're using a WebDriver-based testing tool. We'll use this orb to install the latest versions of the browsers we want to use during testing.
With our basic configuration out of the way, we can move on to the main section of any CI service - running a job.
Configure CircleCI to run TestCafe tests using Google Chrome
The following setting we'll configure inside our .circleci/config.yml file is the jobs setting. This setting consists of one or more sections where you define the environment where you want to run your build and the steps to be performed. To start, we'll create a job to run our TestCafe tests using Google Chrome on headless mode. The configuration to get it to work looks like this:
version: 2.1
orbs:
node: circleci/[email protected]
browser-tools: circleci/[email protected]
jobs:
chrome:
docker:
- image: cimg/node:12.22-browsers
steps:
- checkout
- browser-tools/install-chrome
- node/install-packages
- run:
name: Run TestCafe tests on headless Chrome
command: npm run test:chrome:headlessThe first key under the jobs setting is the name of this specific job. We called this job chrome, but it can be any name you prefer. It's best to use a descriptive name here since we'll use it later in the configuration file. Under the name of your job, you'll begin defining the specifications on how to run the test. In this scenario, we have two settings to configure.
First, we have to tell CircleCI which environment to use, also known as an executor. As mentioned earlier, CircleCI allows you to execute your builds on different systems like Linux, Windows, and macOS. The most common option to use on CircleCI will be a Linux-based environment running from a Docker image, which we can configure using the docker setting.
The docker key requires the image setting to let CircleCI know which Docker image to use for the test environment. CircleCI provides pre-built Docker images that work great on their service. For a Node.js-based application, we can use CircleCI's Node.js Docker image, which has most of what's necessary to run a TestCafe test suite.
In this instance, the Docker image will use Node.js version 12.22 - the Node.js version initially used while building the test suite. CircleCI maintains Docker images for almost all new versions of Docker, so in the future, we can easily change the image name if we need to use a different version of Node.js. We'll also use the "browsers" variant of the image, which also contains the necessary tools to set up our browsers later in the configuration.
After configuring our build environment, it's now time to define what actions to take inside that environment using the steps setting. This section allows you to set a series of steps, either a pre-defined string provided by CircleCI for an orb or a map to run any command you need. Here, we have four steps:
checkout: This string is a special step provided by CircleCI to place your project's source code inside the build environment. In our case, it will copy all of our TestCafe tests inside of the Docker image for later use.browser-tools/install-chrome: This command comes from thecircleci/browser-toolsorb and executes multiple commands under the hood. The command will install the latest version of Google Chrome inside the Docker image to have it available for our tests.node/install-packages: Another command from an orb, this time from thecircleci/nodeorb. The command will install all of the dependencies from a Node.js project as efficiently as possible, like restoring cache from a previous build and only installing any new dependencies.run: The final command is to execute our TestCafe tests once our build environment gets set up with Chrome and TestCafe dependencies installed. This step defines a map with two keys - an optionalnameto describe the test and acommandto execute inside of the Docker image.
The command to run the tests in the final step is npm run test:chrome:headless, defined as a script inside the project's package.json file. The script uses the installed TestCafe dependency and runs all test files from the repository on headless Chrome.
We've defined our build environment and test steps, but we still need one piece of information to tell CircleCI our workflow for running a build. A workflow is our way to configure which jobs we want to run and how they should get executed when triggering a build. To define a workflow, CircleCI uses the workflows setting, as shown below:
version: 2.1
orbs:
node: circleci/[email protected]
browser-tools: circleci/[email protected]
jobs:
chrome:
docker:
- image: cimg/node:12.22-browsers
steps:
- checkout
- browser-tools/install-chrome
- node/install-packages
- run:
name: Run TestCafe tests on headless Chrome
command: npm run test:chrome:headless
workflows:
end-to-end-tests:
jobs:
- chromeThe first key under workflows is the name of your workflow, and under this setting, you'll define your jobs using the jobs key. CircleCI workflows let you configure many different options for your continuous integration needs, like scheduled jobs or jobs requiring manual approval from a team member. For our scenario, we only need to trigger a single job, so we only need to set the name of the job we want to run (chrome).
With a fully-fledged .circleci/config.yml file all set up, place it inside of the code repository and commit it. Once the configuration file is in place, you can trigger a build on CircleCI by going to your project in the CircleCI service and choosing the "Use Existing Config" option. CircleCI will connect to your code repository, fetch the configuration, and begin performing each defined step in your job.
If all goes well, you'll have your first test suite executed and a friendly green passing status:
This simple example is just a taste of the ease that CircleCI provides for your testing. With a short configuration file, you're able to use CircleCI to spin up an entire Linux environment using Docker, install Chrome and Node.js dependencies using the orbs functionality, and have everything ready to execute your test suite. Now that the initial setup is out of the way let's see how simple it is to run tests under other browsers on CircleCI.
Configure CircleCI to run TestCafe tests using Mozilla Firefox
Let's add another job to run our tests, this time using Mozilla Firefox. We already have most of the setup required by CircleCI, like specifying the CircleCI version and which orbs we want to use. The only thing we'll need to do is add a new section under the jobs setting and make sure to execute it in our workflow:
version: 2.1
orbs:
node: circleci/[email protected]
browser-tools: circleci/[email protected]
jobs:
chrome:
docker:
- image: cimg/node:12.22-browsers
steps:
- checkout
- browser-tools/install-chrome
- node/install-packages
- run:
name: Run TestCafe tests on headless Chrome
command: npm run test:chrome:headless
firefox:
docker:
- image: cimg/node:12.22-browsers
steps:
- checkout
- browser-tools/install-firefox
- node/install-packages
- run:
name: Run TestCafe tests on headless Firefox
command: npm run test:firefox:headless
workflows:
end-to-end-tests:
jobs:
- chrome
- firefoxOur new job is called firefox, and the setup looks almost identical to the Google Chrome job. We're setting up the same environment with CircleCI's Node.js Docker image and perform similar steps. We're still using the special checkout step to place our tests inside the build environment. Then, we'll use the browser-tools/install-firefox command from the circleci/browser-tools orb to install Firefox. TestCafe's dependencies will get installed using the node/install-packages command from the circleci/node orb. Finally, we run our tests using the command set up to run tests on headless Firefox.
Since CircleCI already knows about our configuration and has hooked up with the code repository, all you need to do is update the .circleci/config.yml configuration file in your project and commit it to the repo. CircleCI will pick up on the code commit and immediately trigger a new build with the additional job.
Unfortunately, these changes will cause your tests to fail:
What happened? In our case, the reason these tests now fail is that they're running at the same time. When setting more than one job in a CircleCI workflow, all jobs run in parallel by default. Unfortunately, that won't work well with our particular test suite. The TestCafe tests are pointing to the same application, inserting and deleting data simultaneously, so they'll eventually trip over each other. For instance, the Chrome job may delete data from one section just before the Firefox job runs an assertion looking for something previously set up in the test, causing it to fail.
Ideally, we would isolate the data for each continuous integration job. Setting up data for end-to-end tests to fix issues like these takes some effort, but that's out of the scope of this article. Instead, we'll tell CircleCI to run each test sequentially instead of concurrently to avoid these problems. To accomplish this, we can tell CircleCI that the firefox job requires the chrome job to finish:
workflows:
end-to-end-tests:
jobs:
- chrome
- firefox:
requires:
- chromeThe requires key under a job name tells CircleCI to wait until all jobs defined under it have finished building, regardless of the results. By making this change, CircleCI first runs the TestCafe test suite using Chrome, and once done, it will immediately trigger the test suite using Firefox. While this means our total build time will double, it will solve our data issues and make our tests pass on both browsers.
Configure CircleCI to run TestCafe tests using Microsoft Edge
Next up, we'll add another job to run the test suite using Microsoft Edge. CircleCI has Windows-based environments to run your tests. However, these build environments mainly target Windows-specific projects, such as building Universal Windows Platform apps or projects using the .NET framework. It does not have Microsoft Edge installed, and there's no simple way of setting it up in that environment.
Luckily, Microsoft provides a Linux-based installation of the beta and development versions of Microsoft Edge. The Docker image we've been using for the Chrome and Firefox jobs is based on Ubuntu Linux, so that means we can install Microsoft Edge on the build environment before executing our end-to-end tests.
As we did in the previous section, we're going to add a new job definition after the Firefox job setup in .circleci/config.yml called edge for running the TestCafe test suite using Microsoft Edge:
edge:
docker:
- image: cimg/node:12.22-browsers
steps:
- checkout
- node/install-packages
- run:
name: Set up Microsoft Edge
command: |
curl https://packages.microsoft.com/keys/microsoft.asc | gpg --dearmor > microsoft.gpg
sudo install -o root -g root -m 644 microsoft.gpg /etc/apt/trusted.gpg.d/
sudo sh -c 'echo "deb [arch=amd64] https://packages.microsoft.com/repos/edge stable main" > /etc/apt/sources.list.d/microsoft-edge-dev.list'
sudo apt update
sudo apt install microsoft-edge-beta
- run:
name: Run TestCafe tests on Microsoft Edge
command: npm run test:edge:headlessSetting up the build environment is identical since we'll use the same Docker image. However, the steps need to change a bit with some extra setup for Microsoft Edge. We're still using the checkout step and installing dependencies using node/install-packages. But the CircleCI orb command for installing browsers won't get used here since it can't install Microsoft Edge.
Instead, we'll manually follow the command line installation instructions from Microsoft's website as a step just before running our tests. These commands will set up the Microsoft Edge repository for Ubuntu-based Linux distributions and install a binary of the latest beta version of the browser. Once the browser is installed, we can run our TestCafe tests on headless Edge, just like we do on Chrome and Firefox.
The last step is to include the edge job in the workflow, making sure to run the job sequentially instead of in parallel with another job. The end result of our CircleCI configuration file will look like this:
version: 2.1
orbs:
node: circleci/[email protected]
browser-tools: circleci/[email protected]
jobs:
chrome:
docker:
- image: cimg/node:12.22-browsers
steps:
- checkout
- browser-tools/install-chrome
- node/install-packages
- run:
name: Run TestCafe tests on headless Chrome
command: npm run test:chrome:headless
firefox:
docker:
- image: cimg/node:12.22-browsers
steps:
- checkout
- browser-tools/install-firefox
- node/install-packages
- run:
name: Run TestCafe tests on headless Firefox
command: npm run test:firefox:headless
edge:
docker:
- image: cimg/node:12.22-browsers
steps:
- checkout
- node/install-packages
- run:
name: Set up Microsoft Edge
command: |
curl https://packages.microsoft.com/keys/microsoft.asc | gpg --dearmor > microsoft.gpg
sudo install -o root -g root -m 644 microsoft.gpg /etc/apt/trusted.gpg.d/
sudo sh -c 'echo "deb [arch=amd64] https://packages.microsoft.com/repos/edge stable main" > /etc/apt/sources.list.d/microsoft-edge-dev.list'
sudo apt update
sudo apt install microsoft-edge-beta
- run:
name: Run TestCafe tests on Microsoft Edge
command: npm run test:edge:headless
workflows:
end-to-end-tests:
jobs:
- chrome
- firefox:
requires:
- chrome
- edge:
requires:
- firefoxConfigure CircleCI to run TestCafe tests using Apple Safari
To complete the full coverage of most major browsers for our end-to-end tests, I intended to set up a CircleCI job to run the test suite using Apple Safari. Since CircleCI provides macOS build environments, it sounded like it was possible. Sadly, I couldn't get this working at all using CircleCI's included environments or tools.
CircleCI offers different versions of macOS, including 10.13 (High Sierra), 10.14 (Mojave), and 10.15 (Catalina). However, only some of the more recent macOS Catalina builds on CircleCI have the necessary support for UI testing by disabling MacOS's System Integrity Protection. The older builds provided by CircleCI have System Integrity Protection enabled, making it impossible to run TestCafe.
On top of the System Integrity Protection setup, there's also a long-standing issue with TestCafe on macOS Catalina where it needs Screen Recording permissions to execute even if you don't use screen recording. Since you can't manually accept these permissions from a headless CI system, trying to run the tests without the proper permissions from macOS will fail.
In the end, it wasn't possible to use CircleCI to run tests on Apple Safari. If you use CircleCI as your continuous integration service and need to run end-to-end tests using Apple Safari, you'll need to use an external service like BrowserStack or LambdaTest.
Summary
CircleCI is an easy-to-use continuous integration service, yet highly flexible and powerful to scale up with your projects. It provides plenty of options for running your tests in different build environments, so you will likely find the proper setup for your needs. All you need to get started is a GitHub or Bitbucket account since it has a generous free tier to begin building your first project quickly.
This article shows how easy it is to use CircleCI to run your TestCafe tests using Google Chrome, Mozilla Firefox, and Microsoft Edge without resorting to third-party services. You can spin up multiple build environments based on Linux through pre-built Docker images in a short and straightforward configuration file. Thanks to CircleCI orbs, all you need is a single line of setup to install a browser or set up dependencies. Finally, you can orchestrate how your jobs run with a flexible workflow system.
Although we couldn't use CircleCI to run TestCafe tests using Safari, CircleCI is still an excellent choice for running complete end-to-end tests for web applications. It's one of the better continuous integration to ensure your app is working as expected after every change.
Have you used CircleCI for running end-to-end tests for web applications? If you have, share any tips, tricks, or personal experiences with others in the comments section!
