Are you part of a QA team with a comprehensive set of automated tests providing plenty of coverage? Consider yourself fortunate since few organizations dedicate the time and effort needed to build and execute a robust automated testing system.

However, are you part of a team that has a well-thought-out strategy for building and executing automated tests? That's more of a rarity than you might expect.

In my experience, most companies who begin to delve into automated tests start working on them as if they were stockpiling for a drought. The team builds, builds, and builds some more, thinking that more tests and test runs will translate into higher quality. Eventually, they'll reach a point where thousands of automated test cases become a distraction to their work instead of a valuable part of their processes.

Instead of piling up automated tests, teams should spend time figuring out which tests to run and when. That way, teams get the most out of their efforts. More importantly, it reduces the chance of test automation a roadblock with other people's work. This article discusses why running automated tests without a strategy is a recipe for disaster and offers guidance to help testers and QA teams make the most of their current and future test suites.

The Pitfalls of Running Automated Tests Without a Strategy

Doing anything without a strategy is an almost sure-fire way to run straight into failure. If you're starting a business, you won't survive for long without planning for the inevitable obstacles and difficulties that will come along your path. If you want to lose weight and get in shape, there's a good chance you won't accomplish your goals without thinking about how to approach exercise and diet. The same goes for running automated tests. Sure, you might get lucky and have good results. But I guarantee you'll have a much easier time with a strategy in place.

One of the most common issues with building and executing tests without planning is the time it takes to run them. When a team starts making automated tests, running them typically takes just a few seconds. But every new test case, even if it takes less than a second to execute, eventually adds to the total time needed to run through them. The slower testing times happen incrementally, and it's difficult to fix the issue by the time teams realize what's happening. It's like death by a thousand tests.

Another typical problem is test flakiness. The more tests you add to a test suite, the more you'll see inconsistent results. It's an unfortunate side effect of test automation. Debugging a flaky test suite is also significantly more challenging when you have hundreds or thousands of automated test scenarios since there's a high chance you can't replicate it easily. Flakiness eventually leads to testers wasting time looking for false positives or negatives or—worse yet—making teams give up on automated testing altogether.

The root of these issues isn't that the test suite contains too many automated tests, although that's possible if the team focuses on vanity metrics like the number of test cases they have. Usually, the problem stems from running the tests at unsuitable times throughout the development lifecycle. Just because you have a battery of automated tests you can execute at any time doesn't mean you need to run them all the time.

A Painful Example of Running Automated Tests Without a Strategy

One example that comes to mind is a team I worked with, where they had an exhaustive set of UI tests for their web application. The test run took over 30 minutes to complete. I noticed that the team set up these automated UI tests (and other automated tests) to run every time a pull request was opened or updated on their GitHub account. The problem was magnified the closer the team was to their deadlines when more commits and revisions occurred.

Even worse, the team leads configured their GitHub repo to prevent developers from merging code until all tests passed. Given that these were UI tests, it meant there were plenty of tests failing sporadically for no reason, and the team often had to wait for over an hour to be able to merge even the smallest of code changes. As you might expect, this process was a massive pain point for the team.

Eventually, they removed the restriction of needing a green test suite to merge code, eliminating the bottleneck but leading to worse problems. They didn't address what I thought were more significant issues—the slow-running UI tests and their flakiness. That meant the team began merging in buggy code that caused countless regressions because they didn't wait or trust their automated test suite.

Fixing a flaky test suite is a separate topic we won't get into for this article. Still, the team could have avoided a large portion of these issues if they had thought of a strategy for running these tests early in the process. Instead of creating more and more tests, figuring out how to execute them at the right time goes a long way in test automation effectiveness.

A Practical Strategy for Running Automated Tests Effectively

For test automation, a valuable strategy for executing your test suite is gaining enough understanding of the different types of automated tests you have and determining when's the best time to run them so they don't disrupt the team. Most companies with automated testing have various tests to run, and almost all of them just run them whenever possible. That's where they typically run into problems as more testing happens.

In this section, I'll talk about a common strategy I've successfully used to help organizations make the most out of their automated tests according to their type.

Static code analysis

Although some developers don't consider static code analysis like linting or type validation as testing, it's an essential part of ensuring a healthy codebase since it catches typos, detects incorrect usage of variables and functions, and maintains consistency with code style guidelines. Implementing static code analysis is also critical because it should occur on the frontlines—right when developers are coding.

Static code analysis should be integrated into the team's development workflow and run without manual intervention. Modern code editors can do this analysis on demand as developers type their code by automatically linting the code and alerting about typos and other errors. Git hooks can do the same process by running a quick check before committing new code or pushing it into the remote repository. These checks only take a few seconds, so there's no excuse to run them as much as possible.

If your team doesn't do any static code analysis, it should become a priority if you want to have a solid codebase in the long run. Integrating static code analysis tools for older codebases can be tricky since you'll likely have hundreds or thousands of violations. Most analysis tools will let you configure them to temporarily ignore existing violations to give the team a chance to correct them while working on new code. It can be a lot of work, but the payoff is worth it.

Unit tests

Since unit tests are typically quick to execute and have little to no dependencies around the codebase, they should also run as much as possible by triggering them manually or automatically. Developers should run these tests locally as they modify their code to ensure no regressions have happened with their changes before pushing them to the code repository. Fixing bugs is much cheaper to do the quicker it's discovered, and unit tests can rapidly uncover defects that can help developers patch things up before pushing out their code to the rest of the team.

A functional approach for automatically running unit tests is executing them in a continuous integration system when opening or updating a branch for a code review. Services like GitHub and GitLab can show the status of unit tests right in the pull request or merge request, so the team reviewing the code knows that the new updates haven't caused any regressions. Teams should also run their unit tests when updating specific branches in the repository, like when the main branch receives new code.

One complaint I've heard from teams who don't run unit tests as often as they should is that they run too slowly. If this is the case, you have a few options. An easy way to cut down on execution time is to run unit tests in parallel, which most testing tools support. If running unit tests in parallel isn't an option because they rely on a specific order, consider reviewing and refactoring your tests since they ideally should have no ordering dependencies and run independently.

Integration tests

When digging into integration tests, it becomes trickier to determine when it's best to execute them. Building and running automated integration testing for monolithic applications is usually straightforward, ensuring that each of the system's different components work as expected with each other. Running integration tests alongside unit tests can work well, especially for smaller monoliths.

However, integration testing becomes more complex and critical for applications tightly coupled with other services or built on microservices. Modern architectures with multiple microservices make executing integration tests challenging, even though they're essential for validating multi-tiered applications.

Larger applications present unique challenges in integration testing, such as data management and setting up test environments. While these tests are crucial, it might be infeasible to run them frequently. Using strategies like mocks can help, but they need maintenance. Ideally, if your application heavily relies on external services, it's vital to establish a robust foundation for executing integration tests throughout the software development lifecycle.

End-to-end tests

Like integration tests, end-to-end testing is crucial for software QA, especially for modern applications that need external systems like retrieving data from a third-party API or pushing long-running tasks to a queue for later processing. The challenge with end-to-end tests lies in their instability and slowness, making them time-consuming and prone to failure. Many development teams shun end-to-end testing because of these reasons. While no one wants slow-running or flaky tests bogging down the team, not having these tests is a missed opportunity to improve the overall quality of the application.

My preferred strategy for running automated end-to-end tests is to execute a smaller subset of these as a smoke test during code reviews and run the entire test suite only when merging code into one of the main branches of the repository. The smoke tests should consist of a handful of test scenarios to ensure the code changes in a branch haven't broken critical functionality, and they need to run as quickly as possible—or at least fast enough that it doesn't raise complaints from the team.

Another strategy I encourage teams with extensive automated end-to-end tests is to leverage their continuous integration systems to run them during off-peak hours. CI systems have the functionality to schedule these types of jobs, so you can automatically trigger the test suite when it doesn't disrupt the rest of the team and have the CI service send a notification of the results. This process ensures you use the tests you've built with minimal interruptions.

Performance tests

There's a growing emphasis on performance testing for all types of applications, and I'm all for it. Verifying an application's responsiveness, stability, and scalability will let teams handle increased usage of their systems with fewer growing pains. While many applications might not need to monitor performance metrics closely due to their existing use, it provides a crucial quality element for both the present and future. Executing performance tests requires careful planning to yield the best outcomes.

While testing against production provides the most accurate results, it risks disrupting their customers, especially if the system isn't prepared for these tests. An alternative is to run performance tests in a separate environment. These days you can easily spin up a complete test environment that's a near-perfect replica of production, although the cost and effort to maintain this might be infeasible for some teams. Regardless of the environment, running automated performance tests should occur infrequently, like when primary branches receive updates or on a set schedule. This frequency is often enough to give you the data you need to validate the performance of the underlying application. Unless performance is of the utmost importance for the application, there's rarely a need to run performance tests constantly.

Most organizations looking to spend more time building automated performance tests should first invest more time in monitoring and observability instead. A solid platform that keeps track of the real-time health of your staging and production environments is usually enough to know about performance regressions when they occur. Combine these systems with a sprinkle of automated performance tests that help pinpoint trouble spots, and you'll have all the coverage needed to keep your application humming.

Building Your Unique Strategy for Improved Test Automation Execution

The strategies for the different types of automated tests mentioned above aren't a "one-size-fits-all" solution, especially since plenty of other automated testing methods are not covered here. The examples provided in this article can guide you to find improvements to your existing processes, but the strategies you develop for your systems depend on countless factors. The application architecture (like running a monolith or microservices), the project structure (like having a monorepo or multiple repos), and the required infrastructure needed to execute each test all come into play when deciding how to get the most out of your test runs.

The key to finding your ideal strategy is to find the balance between getting the information you need out of your automated tests and not slowing down the team while you do that. This balance is critical because you'll want to get the most out of your test automation processes, but if it's disrupting the development process, the value of automation decreases significantly. Keeping your automated tests running efficiently ensures the team can provide the highest quality possible for their applications.

Running automated tests without a short- or long-term strategy can lead to many problems that hinder the software development process. A carefully considered approach will increase the power that automated testing provides when done right, and it's well worth the time and effort you put into it.

What automated testing strategies have you and your team found effective? Leave a comment below and share your techniques with the development and testing communities!