Does your development team groan in agony each time they deploy because regression testing takes forever to complete? Do testers despise having to fix the regression test suite when it breaks because it takes forever to verify that everything's working? Is the entire team going on coffee runs or lunch breaks because they know it'll be a while until they get any reports from your regression tests?
If any of those apply to you, these are clear signs that your regression tests are bloated and can use some tender loving care as soon as possible.
Regression testing is just one portion of a balanced plan alongside other forms of testing, like unit testing and exploratory testing. However, it's an essential key to any healthy software product. It ensures that any new changes to the application under test haven't affected critical parts of the system.
When done right, regression tests reduce the risk of problems creeping into a previously working system. In turn, the tests increase the confidence of the entire team when delivering a final product. It's a win-win for everyone on the project.
The value of a regression test suite becomes more apparent during the project's lifecycle. The project will eventually become more complex, needing more coverage of regression testing. Having more coverage is excellent, but it can become a problem. The more tests you have, the more bloated the test suite becomes. It can quickly reach a point where the test suite becomes so large, and it becomes far less useful to run.
When a test suite becomes too large, the effectiveness of the tests becomes significantly minimized. For instance, if the team has a delivery pipeline that includes ensuring regression testing passed, extended run times can hurt deployments. Waiting around for tests to pass also affect the testers maintaining the test suite. Finding a broken test in the middle of a broad suite can get tricky. You won't know for sure if it's an isolated incident or a wide-spread problem until you run everything.
So, how can your testing team avoid these problems?
Set a cap on the amount of time it takes to run your suite
The best way to avoid your regression test suite from expanding beyond control is to set the maximum amount of time allowed to run the entire suite.
By specifying ahead of time how long you want your tests to run, you'll avoid all of the problems that come with a long-running test suite. If you have a cap, your team will know with certainty how long the regression testing phase should take. Knowing this will allow you to plan your delivery schedule better. Anything more than the allotted time permitted is a problem that the team must address as soon as possible.
The amount of time you should set is dependent on several factors. It depends on the application's complexity, the number of test cases required to provide confidence, your team's skills, and more. I can't offer a specific amount of time because everyone has different needs for their project. I've seen teams decide on times anywhere from as low as 30 minutes up to 24 hours.
Get together with all the stakeholders involved and come with an agreement on this. Developers, testers, and product managers need to have a vested interest in this issue. Everyone should understand the importance of having a balance between running the right tests and how long it should take to provide the confidence needed to forge ahead.
The shorter the execution time, the better it would be. Faster feedback to handle issues that arise is ideal. But you also need to ensure that the tests are checking for the right things, no matter how long it takes. Make sure everyone is clear on this distinction - it's not all about time.
What to do when your tests go over the limit?
When you're beginning to build a test suite, it's easy to ensure that your suite is sleek and speedy. But as the team adds more tests, the amount of time it will take to run the regressions increases.
Inevitably, there will come a time when your regression tests reach the point where they're too slow, too much of a hassle to deal with it. When this happens, your team needs to get together to figure out how to handle this.
Here are three of the most common approaches that teams use to improve their test execution times:
Approach #1: Improve the performance of the tests or the application under test
Taking a good, hard look at how your tests are written and executed may help with performance. Sometimes tests can be rewritten to run specific scenarios more efficiently. Often, this happens with less experienced test automation engineers or new team members who don't know the ins and outs of the project yet. Spending time on refactoring in these situations can yield decent results.
It's not all on the testers, though. The development team for your project should also be in the loop. There's always the possibility of code changes that accidentally tank the overall performance of the application. Memory leaks can degrade performance over time, surfacing only in specific scenarios. A single query that works in development can grind the app to a halt when there's more than a handful of database records. If you suspect that there's a performance regression surfacing in your tests, let the development team know so they can verify.
While occasionally revising your test code is useful, this approach often has the least return on investment. Refactoring code offers no guarantees that you'll make a dent in improving the regression test suite. It takes time and effort for potentially little or no return. Use this approach only if you're confident that some test cases need refactoring.
Approach #2: Run a subset of your regression tests during the workday
Depending on the size of your test suite, you might not need to execute all of the tests for a successful regression test run. If you can extract some of the most critical tests in the suite that covers a good portion of the overall system, you can significantly speed up the test execution.
One way I've seen this approach work is by separating the main regression tests into smoke tests. The smoke tests are a small percentage of the entire regression test suite - usually less than half of the total test cases. The team ran the smoke tests whenever they needed to in their workflow, like after tagging a release for deployment. The entire test suite was then configured to run during off-hours, often in the middle of the night.
However, consider it a red flag if a team is running this approach. It's likely a sign that there are too many tests that aren't useful in the test suite. Typically, this comes in the form of repetitive tests. I've seen test suites that include test cases going through the same functionality except a small change or two. For example, the test fills out the same form but selecting a different radio button, with the same test result. In cases like these, you don't need to go through all of these permutations.
Approach #3: Prune your regression test suite
Your best bet in keeping your test execution time low is to eliminate less-useful test cases aggressively. In the previous two approaches, I mentioned that there could be test cases that aren't useful tests. It can be a poorly written test that's not valuable, or a repetitive test case, or a test case that's no longer valid. These types of tests are great candidates to remove from the suite.
Admittedly, taking this approach is easier said than done. You can't just remove any old test and call it a day. You need to consider the test's usefulness carefully. If the test is poorly written but is the only test that verifies a critical part of the system, you should keep it after refactoring. If a test seems repetitive, check that the permutations it's going through are necessary.
There's also a more subtle issue that surfaces occasionally. For some testers and developers, letting go of code is hard. Some people won't understand why you're throwing away what seems like a perfectly good test. Others experience anxiety, thinking that part of the safety net they created around tests is gone forever. This behavior is especially evident in junior team members. If you find someone on the team going through this, reassure them of the benefits of pruning the codebase.
This approach isn't a one-time task that you do once and never go through it again. You need to do this consistently to keep your test suite healthy. Make sure you're putting time regularly to go through the regression test suite and determine what should stay and what goes.
Keep your promise
Once you have your regression test suite under control, keep it that way. Do not waver from your agreement.
Treat your maximum allowed execution time like a service-level agreement (SLA) your company has with another business or customer. Just as an SLA is a commitment of service, quality, and responsibility, your capped time agreement is a commitment to keeping the test suite running smoothly. Establish a team culture where if the SLA breaks, it needs to get fixed - no questions asked.
There are times when you'll need to reassess the previously agreed capped amount of time. If the project grows significantly with new features and things to test, you'll have to end up with a more extensive regression test suite. Get together with the team once again when this happens. Similar to renegotiating an SLA that has outlived its usefulness, adjust as necessary to fit your needs.
Just make sure you're not allowing a longer execution time because it's the easy road to take. I've seen many teams do this and always end up regretting their decision when their test suite becomes unwieldy.
Conclusion
Keeping a speedy and useful regression suite will benefit the entire team. Any of the approaches mentioned work well. A combination of different methods can be more effective than a single path alone.
Whatever your approach is, always lean on maintaining the quality of the overall test suite. It's nice to keep the tests running smoothly and efficiently, but none of that matters if no one trusts that you're testing the right things.
Get together with your team and let them know about the importance of having a sleek regression test suite. Together, you'll be able to keep a speedy and useful regression suite. It will surely benefit the entire team and maintain both the health of your test suite and the application under test in optimal condition.
Photo Credit: Veri Ivanova on Unsplash
