A few months ago on Reddit, I stumbled upon a thread titled "What does successful automation look like to you? Have you ever seen it?" The post was made by an SDET, explaining their experience across two companies where their automation efforts repeatedly stalled for various reasons, from competing priorities to pivots in the company's business model. The author of the post wanted to know if anyone has actually accomplished a functional and well-maintained test automation suite and glean some insights as to what it takes to get to that point.
Unfortunately, the original poster's experience is a fairly common one on many software development teams. Software testing usually takes a back seat in the list of business priorities and is often the first thing pushed aside when there's a time crunch. Testing is regularly not even part of that list and is something that's tacked on to the development schedule after something blew up spectacularly in production.
That doesn't mean that any business that builds software is cutting corners when it comes to quality. There are plenty of companies that give software testing the proper attention it deserves and know how important it is. However, does it mean that their automation processes are successful? As with most replies to questions related to software development and testing, it depends.
What Is Success in Test Automation, Anyway?
Just because a team spends enough time building their automated testing processes, it doesn't mean that it's a success. You've likely seen your fair share of test automation going wrong despite a heavy investment in people and time. Flaky tests, continuous integration runs that take forever, and bugs still slipping through to production are just a few places that can completely destroy any efforts to build a working automation process.
However, figuring out whether these automation processes are a success requires more than the clearly evident problems of slow, inconsistent results. Occasionally we think that our test automation works well because it runs fast and produces consistent results. But does that mean we should have full confidence that the test automation is helping? Not necessarily. The tangible issues we see, like flakiness and bugs in production, are one part of the equation. However, we also need to ensure other areas, such as that test automation is helping the team while they're in the middle of development.
As an example, one technique I often use to verify if a project's test automation processes are helping out is to deliberately introduce subtle bugs into some critical segments of the codebase. For instance, I slightly change the result to be off by one or invert some boolean logic. If the automated test runs detect the bugs and tell me where the issue is so I can fix it quickly, it helps boost my confidence in the test suite. It's a quick and dirty form of mutation testing that can be done in minutes, and this trick has helped me smoke out test automation implementations that seem to work on the surface but are actually hampering the team's efforts.
Signs That Your Automation Is on the Path to Success
A successful test automation implementation varies from team to team and from project to project. For one team, success might mean having fast CI pipelines that allow them to continuously deploy their app to production. For another, it might mean developers feel fully confident about the test results and their ability to resolve issues as soon as a failure arises.
There isn't a specific metric that fits all teams, but there are a few signals that indicate they're on the track to building successful test automation processes. Here are three worth paying attention to.
Sign #1: The team is delivering more value faster
Many teams pay close attention to indicators such as fewer bugs or faster feedback loops on code changes when trying to measure the success of their automated testing. While these are good metrics to keep an eye out for, they're often treated as vanity metrics internally. The real purpose of having fewer defects and shorter development cycles is that the increase in velocity leads to more value being delivered to the users of your app.
The most tangible sign of success with automated testing is when the software development lifecycle seems to flow effortlessly. New features can ship regularly, if not daily. Developers push updates to the codebase and can know with full confidence if their changes are ready to deploy. There are fewer hotfixes to make, fewer deploys to roll back, and fewer arguments about whether or not to ship on Fridays. Each of these on their own can signify success, but the real reason we build software is to provide value, and a successful test automation implementation makes that much easier.
Sign #2: Coverage goes beyond code
Another metric often deployed by teams looking to justify their test automation is code coverage. Unfortunately, this metric is frequently the wrong thing to measure. I've seen automated test suites going over 95% of a project's codebase but still riddled with bugs because the tests only verified that the code ran, not that it behaved as expected. Using code coverage as one of the primary ways to measure effectiveness only leads to low-value, low-quality tests that usually exist only to artificially boost up those numbers.
In my experience, the teams with the most success with their automation strategies still track code coverage, but they go way beyond using it as a valid metric. Instead, they're looking at the bigger picture by thinking about coverage in other areas that lead to a healthy application. How does the system perform in the real world under heavy loads? Do the most critical business paths work from start to finish? Are there any security vulnerabilities that can expose sensitive customer details? A successful test automation process is a balanced one, incorporating unit tests, end-to-end flows, performance testing, security scans, and more. This type of coverage is what leads to higher quality and faster delivery.
Sign #3: There's a clear dedication to the cause
Sadly, most software teams regard testing as a second-class citizen, whether consciously or subconsciously. It's rarely discussed as part of the current development cycle or sprint, test run results get ignored, and it's frequently the first thing out the window whenever there's a risk of the schedule slipping. In these environments, test automation can never thrive and become a successful part of the process. If test automation is treated as the so-called "red-headed stepchild" of the development lifecycle, it will never find success.
The only way test automation can prosper is when the team commits to the cause by giving it the time and effort it deserves. When flaky tests appear, they get investigated and fixed as soon as possible instead of being cast aside. Developers and testers ensure that any bugs that snuck into production have test coverage to make sure they don't appear again. The code that powers automated tests gets frequent reviews and refactoring in the same way as the code that powers the app. There's never any discussion about skipping tests to rush through a release. All of these acts combine to make test automation succeed instead of having it bolted on as an afterthought.
Wrap Up
Test automation succeeds when it helps ship your project easily, instead of making the deployment process full of worry and dread. Getting to that point requires more than taking the time to build a robust test suite. It also needs to come from an environment that cares about maintaining the long-term health of the automated tests, where issues are detected and fixed quickly and the team has full confidence in their results. It also comes from understanding that testing needs to cover multiple facets.
If your current test automation implementation feels like it's working against you and your team, start by looking at whether your tests help deliver value, whether by making development effortless or delivering a high-quality application into the hands of customers fast. Also, focus on maintaining that high level of quality by ensuring you have test coverage beyond the basics and by fostering a culture of testing across the team. When all these elements align, test automation will become the success it was always meant to be in your organization.
