Doing load and performance tests on your web applications is becoming more critical than ever in today's connected world. People are becoming less tolerant of sluggish experiences online due to high-speed connectivity available almost everywhere globally. With more and more people doing their work at home, coffee shops, and anywhere else with access to the Internet, having your website work quickly and smoothly is crucial to any organization's success.
Load and performance testing is often done on a backend level, using the APIs that power most web applications nowadays. This process will give you and your team a good idea of how the important sections of your business respond when faced with hundreds or thousands of visitors at any given time. While APIs handle most of the magic that runs a web app, web applications are more than just APIs. There's plenty more to consider when it comes to validating the end-user experience.
Any time someone fires up their web browser and goes to your website, it'll take time to load images, scripts, and more. Over time, these requests can drag down the user's experience. Countless reports have shown that the longer a user has to wait to interact with a website, the more likely they'll leave and never come back. One of the most famous examples is a 2006 report where Amazon said they lose 1% of sales for every 100 milliseconds of latency on their site. I know Jeff Bezos has enough money for many lifetimes, but any organization won't want to lose sales because of a seemingly-insignificant statistic.
Your APIs might be highly-optimized and respond in milliseconds, but that won't matter if the browser spends a long time loading the site. It also won't matter if it appears that it's loading slowly. The ideal solution is to perform realistic load testing that goes through what users go through instead of only hitting the backend APIs. Typically, this is a cumbersome process with few tools that can handle this type of testing at scale and automated. However, it doesn't mean it's impossible. Thanks to the load testing tool Artillery and the excellent testing tool Playwright, you can quickly fire up load tests that simulate real-world usage right in the browser.
Artillery and Playwright: Two excellent testing tools that complement each other well
In a previous article on Dev Tester, I covered how to perform load testing with ease using Artillery. As a quick primer on what Artillery can do, it's a Node.js package targeted towards testing backend systems, like standard HTTP APIs and WebSockets. Besides being ridiculously easy and straightforward to use, it also allows developers and testers to extend its functionality through a plugin interface. When it comes to testing the resilience of your modern web applications, Artillery will help keep your systems performing as best as they can.
On the other hand, Playwright is an end-to-end testing tool rapidly gaining popularity in the test automation world. It allows developers and testers to create end-to-end tests for most browsers with excellent support for modern, dynamic web applications such as auto-waiting for page elements and auto-retries for assertions. It also has libraries for different programming languages like JavaScript, Python, and .Net. While I still reach for TestCafe for my end-to-end testing needs, I've found Playwright to be straightforward and easy to build robust test suites for any web application.
How do these two tools with different testing contexts come into play together? The Artillery team released artillery-engine-playwright, an engine that hooks up with Artillery and allows an Artillery test script to launch end-to-end tests for gathering performance metrics. It's a unique way to do automated performance testing in real browsers.
Your typical Artillery test will gather information about an API's response time. With the Artillery Playwright engine, Artillery will launch headless instances of Google Chrome and measure performance metrics like First Contentful Paint time, Largest Contentful Paint time, and the time it takes for the document to become interactive. All of these are measurements of a website's responsiveness and perceived speed—essential if you want to keep users on your site.
Let's dig into an example to see the basics of how the Artillery Playwright engine works with a simple test script and read its results.
Putting Artillery and Playwright to use
For this example, we'll run a basic end-to-end test using Airport Gap, a website I created to help others practice API testing. For the test, we'll load the home page and click a link to go to a separate page. It's not a particularly helpful test case, but the intent is to measure how fast (or slow) the home page of the website loads and how it handles rendering other pages.
Artillery is a Node.js package, so you'll need to download and install Node.js for your system. You can also use a tool like nvm or asdf to set things up. Once Node.js is set up, you can install the latest versions of Artillery and the Artillery Playwright engine with the following command:
npm install artillery@latest artillery-engine-playwright@latest
The next step is to create a Playwright test script that we'll use for the load test. While the documentation for the Artillery Playwright engine mentions you can use Playwright's test generator to record your actions and create ready-to-use test scripts, I found it to have a few issues (which I'll describe later). Instead, we'll create a quick script manually in a new file called airport-gap-test.js with the following contents:
async function airportsList(page) {
await page.goto('https://airportgap.dev-tester.com/');
await page.locator('a >> text="View Airports"').click();
};
module.exports = { airportsList };
This short test script will use Playwright to load the Airport Gap website, find a link with the text "View Airports", and click the link. As mentioned earlier, it's a simple test script, but it provides an excellent starting point for viewing performance metrics on any given website.
A few things to note about the test script:
- We don't need to explicitly require Playwright in the script itself because the Artillery Playwright engine handles that for us.
- The
pageargument in the function is an instance of Playwright'sPageclass for interacting with the launched browser during test execution. - We have to export the function (
module.exports), which will let us define which function the engine needs to execute in the Artillery test script. You'll see how it works soon.
With our Playwright test, we can now focus on setting up the performance testing side of things using Artillery. We'll create a new file called airport-gap-load-test.yml to build our Artillery test script in YAML format, using the following contents:
config:
target: "https://airportgap.dev-tester.com"
engines:
playwright: {}
processor: "./airport-gap-test.js"
scenarios:
- engine: playwright
flowFunction: "airportsList"
flow: []
At a high level, this test script contains the following details:
- We're telling Artillery to use the Playwright engine in the
config.enginessection. The engine doesn't require additional parameters, so we can pass an empty object in theplaywrightconfiguration. - The Playwright test script gets loaded by Artillery in
config.processor. The Playwright script is in the same directory as the test script in this example. - One thing to note in this example is that we include the URL of the Airport Gap website in
config.target, but it's not necessary to use the same URL that's in the Playwright test script. We still need to include the configuration setup for Artillery to run the test. - Under
scenarios, we're configuring this test to run a flow using the Playwright engine and specifying theflowFunctionto the name of the exported module in the Playwright test script. - Since Artillery runs the test scenarios straight from the Playwright test script,
scenarios.flowwill have an empty array. Likeconfig.target, it's required for Artillery to run.
Now that the Playwright end-to-end test and the Artillery test are set up, we can run them both together through Artillery with the following command:
artillery run airport-gap-load-test.yml
If everything is working correctly, Artillery will launch a headless instance of Chrome and go through the Playwright test script with a single user. After its test run, Artillery prints out the test results:
All VUs finished. Total time: 5 seconds
--------------------------------
Summary report @ 15:17:17(+0900)
--------------------------------
engine.browser.http_requests: .................................................. 12
engine.browser.memory_used_mb:
min: ......................................................................... 2.4
max: ......................................................................... 2.4
median: ...................................................................... 2.3
p95: ......................................................................... 2.3
p99: ......................................................................... 2.3
engine.browser.page.FCP.https://airportgap.dev-tester.com/:
min: ......................................................................... 506.2
max: ......................................................................... 506.2
median: ...................................................................... 507.8
p95: ......................................................................... 507.8
p99: ......................................................................... 507.8
engine.browser.page.LCP.https://airportgap.dev-tester.com/:
min: ......................................................................... 506.2
max: ......................................................................... 506.2
median: ...................................................................... 507.8
p95: ......................................................................... 507.8
p99: ......................................................................... 507.8
engine.browser.page.domcontentloaded: .......................................... 1
engine.browser.page.domcontentloaded.https://airportgap.dev-tester.com/: ....... 1
engine.browser.page.dominteractive:
min: ......................................................................... 389
max: ......................................................................... 389
median: ...................................................................... 391.6
p95: ......................................................................... 391.6
p99: ......................................................................... 391.6
engine.browser.page.dominteractive.https://airportgap.dev-tester.com/:
min: ......................................................................... 389
max: ......................................................................... 389
median: ...................................................................... 391.6
p95: ......................................................................... 391.6
p99: ......................................................................... 391.6
vusers.completed: .............................................................. 1
vusers.created: ................................................................ 1
vusers.created_by_name.0: ...................................................... 1
vusers.failed: ................................................................. 0
vusers.session_length:
min: ......................................................................... 2798.1
max: ......................................................................... 2798.1
median: ...................................................................... 2780
p95: ......................................................................... 2780
p99: ......................................................................... 2780The results provided by the Artillery test run are performance metrics straight from the browser, giving you a glimpse at how long your site takes to load and become interactive. It's a tangible way to measure real-world performance in an automated way.
While running this kind of test as a one-off is okay to get a baseline idea of how your website performs, seeing how your site performs under a heavier load is more beneficial. You can do this easily in Artillery by setting the config.phases configuration in the test script. For example, let's say we want five virtual users to load the website every second for 30 seconds. We can configure the test script by adding the config.phases configuration as specified below:
config:
target: "https://airportgap.dev-tester.com"
engines:
playwright: {}
processor: "./airport-gap-test.js"
phases:
- arrivalRate: 5
duration: 30
scenarios:
- engine: playwright
flowFunction: "airportsList"
flow: []
Running the test in the same way will trigger five headless Chrome instances in your system every second and go through the Playwright test script without having to modify anything else. The results will aggregate and be returned at the end of the complete test run:
All VUs finished. Total time: 33 seconds
--------------------------------
Summary report @ 15:25:53(+0900)
--------------------------------
engine.browser.http_requests: .................................................. 1500
engine.browser.memory_used_mb:
min: ......................................................................... 2.3
max: ......................................................................... 2.9
median: ...................................................................... 2.5
p95: ......................................................................... 2.7
p99: ......................................................................... 2.9
engine.browser.page.FCP.<https://airportgap.dev-tester.com/:>
min: ......................................................................... 399
max: ......................................................................... 784.4
median: ...................................................................... 441.5
p95: ......................................................................... 487.9
p99: ......................................................................... 596
engine.browser.page.LCP.<https://airportgap.dev-tester.com/:>
min: ......................................................................... 399
max: ......................................................................... 784.4
median: ...................................................................... 441.5
p95: ......................................................................... 487.9
p99: ......................................................................... 596
engine.browser.page.domcontentloaded: .......................................... 150
engine.browser.page.domcontentloaded.<https://airportgap.dev-tester.com/:> ....... 150
engine.browser.page.dominteractive:
min: ......................................................................... 315
max: ......................................................................... 701
median: ...................................................................... 340.4
p95: ......................................................................... 424.2
p99: ......................................................................... 518.1
engine.browser.page.dominteractive.<https://airportgap.dev-tester.com/:>
min: ......................................................................... 315
max: ......................................................................... 701
median: ...................................................................... 340.4
p95: ......................................................................... 424.2
p99: ......................................................................... 518.1
vusers.completed: .............................................................. 150
vusers.created: ................................................................ 150
vusers.created_by_name.0: ...................................................... 150
vusers.failed: ................................................................. 0
vusers.session_length:
min: ......................................................................... 1967
max: ......................................................................... 4517
median: ...................................................................... 2515.5
p95: ......................................................................... 3534.1
p99: ......................................................................... 3828.5
Run thousands of browser-based performance tests with Artillery Pro
We executed all of the example test runs above locally. This approach works on a small scale, but your browser-based performance tests will eventually run into constraints. The more tests you run locally, your system will need more resources. Since each virtual user will make the Artillery Playwright engine load a complete instance of Google Chrome, your system will likely run into memory and CPU bottlenecks with dozens of browsers running in the background. It can also produce inaccurate test results if your system struggles to run too many test scripts simultaneously. Even with a powerful workstation, your tests will eventually reach limitations.
If you're serious about load testing your web applications at a scale much more extensive than your local environment permits, you can bring Artillery Pro into the mix. Artillery Pro is a library built on top of Artillery that allows you to run your tests in your AWS account. Using Amazon Elastic Container Service (ECS), you can offload, scale, and execute your load tests in different regions without having to change your existing Artillery test scripts. Instead of worrying if your system can run more than a handful of Chrome instances, you can give this work to AWS and only pay for what you use.
This article won't dive into using Artillery Pro with the Playwright engine—mostly because my humble Airport Gap server can't handle the load. However, the Artillery team covers this topic on their blog. In their article titled "Launching thousands of browsers for fun and profit", they discuss using the largest AWS Fargate containers to run thousands of Artillery tests using Playwright cheaply. It's an interesting read to see the power of using AWS for this type of testing.
Shortcomings with the Artillery Playwright engine
While the Artillery Playwright engine works great to show helpful performance metrics for your web applications, it does have a few shortcomings that I spotted while using it that I'd like to see improved:
- You're limited to using JavaScript with CommonJS to write the tests for Artillery to load the Playwright test script, so you can't take advantage of Playwright's ability to use other programming languages.
- You can't set up assertions in the test script. You can use the Artillery Playwright engine to execute a quick smoke test to ensure the website loads, but its usefulness is limited to your actions in the test script.
- You can't run the test script on its own using Playwright, so you would have to copy the same code in a slightly different format if you're interested in running it as a standalone test script.
- You can't pass any configuration options for Playwright through Artillery. The default settings work well enough, but the ability to change the defaults would help speed up testing. For instance, the default test timeout is 30 seconds, so if something in the test script fails, the test hangs until it times out.
- Error-handling isn't particularly useful for non-developers. If an exception occurs during the test run that isn't a test script failure, Artillery will hang and display a confusing stack trace that isn't too useful for people without a software development background.
- You're limited to headless Chrome, so you can't debug the test script on its own if it begins to fail. And since you can't run the test on its own, as mentioned above, you'll have to duplicate the test and run it separately to have the ability to run it in non-headless mode.
Since the Artillery Playwright engine is meant for performance testing, you'll likely have a few tests similar to the example in this article where you load a page to get some metrics. There's little chance you'll have any of the issues mentioned above in those instances. However, it's always a good idea to know the limitations of your testing tools to help you make better choices for your current situation.
Summary
Performance and load testing are vital for any web application but don't limit it to your backend services only. The frontend—what your users and customers see—is just as crucial as the APIs that power the website. Even if your backend servers and systems are fully optimized, it won't matter if the site feels slow. People will bounce away from your site if it doesn't perform well enough.
Artillery can help you load test your backend APIs, but now it can also go through your entire application stack with Playwright through the Artillery Playwright engine. These tools combined give you a clear idea of how well your website performs and how high load affects the way your site gets served to your users. You can even scale your tests as much as needed through Artillery Pro to push your applications to the limit.
Without realizing it, your organization may be leaving money on the table due to a sluggish performing website. It might feel like your website is running well, but you're just guessing without direct numbers. Using Artillery and Playwright will shine some light on the essential metrics to keep your site snappy and your users happy.
Do you think load testing through the frontend is beneficial for your applications? Leave your thoughts and comments below!
