For the past few weeks, Dev Tester has been covering responses to the question, "How do you know when test automation isn't working?". This question came about from a tweet by the Ministry of Testing.

The tweet spawned lots of replies from testers all over the world. It brought a lot of great insights into issues that we've all dealt with at some point.

For the series finale, we'll tackle an issue that leads to wasted effort and unproductive use of time for any testing team.

Articles in this series

All five articles are also available in a single PDF for free. Just enter your email in the form below to have it sent straight to your inbox.

How Do You Know Test Automation Isn't Working For You Book Cover

Fix some of the most common automated testing issues with this collection of articles from Dev Tester.

Enter your email address to receive a free 37-page PDF with five articles to improve your organization's testing cycle. You will also receive a weekly email with the latest articles posted on Dev Tester.

I know how vital your email privacy is. You will never receive any spam from me, and you can unsubscribe at any time.

The need to manually verify automated tests


Abhishek Roy (@abhishekroy2007) brings up the issue of redundancy when it comes to testing efforts:

At first, I almost disagreed with this tweet. At first glance, I thought the point of this message was to bypass manual testing. I believe that a successful testing approach is to balance different forms of testing.

However, the point I believe Abhishek was trying to make here is when there's redundancy in testing efforts. When automated tests are passing, yet there's an insistence on testing everything automatically, that's a sure-fire sign that test automation isn't working.

For instance, let's say you have an automated test case that logs a user in an application, performs a simple action, and verifies the result. Despite the automated test covering this path, a tester still has to perform the exact sequence of events to check the same result. This kind of redundancy is entirely unnecessary, yet still happens frequently.

If this occurs on your team, there must be a reason why. The first thing that you need to do is dig up the reason why this useless repetition exists in the first place.

Break the habit and balance your testing efforts


One reason this happens is when testing teams introduce test automation when they only have done manual testing. They have spent months or years doing scripted or exploratory testing only, unaccustomed to anything that does part of their work automatically.

In these cases, it's common to see these teams add the extra work of automation to their testing efforts. They don't take time to think about how automation can help free them from testing everything manually. Instead, there's twice the work to do.

When this happens, the team needs to take a step back and create a balance between automation and manual testing. Doing this involves figuring out what's covered or what the plans are for test automation.

If you feel stuck, a good start is to ask the question, "Where will automating these tests help me save time from testing it manually?" It often points you in the right direction.

Once the team has a clear idea of what's automated now or later on, they should focus their manual testing efforts on areas with little or no coverage. Spending time testing areas that don't have automated tests is the best use of the team's time.

However, some teams still don't feel comfortable leaving their work to automated tests. If there's still a strong desire to test those covered areas manually, there's most likely a more deep-rooted issue buried under the surface

Learn to rely on your automated tests


Often, teams experience a lack of trust in their automated test suite. Trust is a recurring theme in this series, but it's also one of the most common roadblocks testing teams face daily.

If there's little or no trust in the automated test suite, the team will always double-check the work. It doesn't matter if your tests cover the entire application's functionality. If there's no trust, no one cares that you have a passing test suite.

If that turns out to be the main issue, an open dialog is once again the key to finding the path forward. Figure out where the lack of trust in test automation is happening. It might take some digging, and you might not like the reasons why. But it's the only way to begin addressing any issues the broader team has.

The problem might be that the team isn't used to automation testing. It can be a lack of experienced testers with enough automation knowledge. It might be that the team has tried to implemented automation testing in the past with disastrous results.

Whatever the reason is, everyone needs to feel confident in the work that the testers are doing. By talking it through and addressing concerns, you can start to tackle the issues with your team. Chip away at any doubts and skepticism about the usefulness of automation.

You can't cover everything, so don't discard manual testing


That's not to say that if you have a robust automated test suite, you should significantly minimize manual testing efforts. Manual testing still has its time and place to use, as long as it's not duplicating something that already exists.

When testing anything, it's nearly impossible to cover all the different permutations for a scenario. You can sit down and write test after test, checking different inputs and outputs. But doing this is not a particularly good use of your time. It also can potentially lead to a slow, unmaintainable test suite that no one wants to run.

These reasons are where manual testing is useful when combined with test automation. Testers can jump in occasionally and test different scenarios for some test cases that are already automated.

It's faster for testers to re-run a test case using different actions and data than to write and maintain different automated tests for each of them. It's especially useful for paths that don't need to get tested after every change in the code.

Performing this kind of manual testing does introduce some redundancy. But it's not a wasted, repeated effort. That's what you need to avoid. As long as you sparingly test automated cases manually with different sets of inputs and assertions, it won't be a fruitless endeavor.

You shouldn't cover all the possibilities with automated testing. You'll end up wasting time and progressively making your test suite worse. You're better off spending that time in other endeavors that bring more value to your testing.

Summary: Combine your testing efforts effectively and avoid duplicating work


Teams have plenty of reasons to spend time doing double the testing work. It shouldn't need to remain that way, however. At best, it's wasting the time of the testing team. At worse, it can make your testing efforts worse.

Spend time to figure out what's the main reason for your team for doing the same tests in different ways. Determining the reason why duplicate work exists helps you shift away from spending too much time on the same work. It takes you from a place of wasting time to balancing your efforts effectively.

One of the main reasons is because teams need to properly transition from a mostly-manual testing approach to a combination of manual and automated testing. As tests are automated, there should be less of a need to test the same scenarios manually.

A great approach to take is to cover high-risk and complex test cases using automation. These are the tests that are critical to the application, or where manual testing is tedious and error-prone. Do just enough automated testing to give your team a high degree of trust and confidence in the test suite.

Covering the high-value sections with automation frees up the team to focus on areas that aren't covered by automated tests. Cover the rest of the application with manual work, like scripted and exploratory testing. Once you demonstrate the benefits of exploratory testing, you'll notice your team reduce their urge to repeat the same tests.

Series conclusion


I can almost guarantee that if your organization has done automation testing, you've gone through some of the issues covered in this series. Maybe you're like me and have experienced all of them at one point or another.

As testers, we all face a standard set of challenges and problems. We might think that our particular problems are unique to our situation. But the chances of having a unique set of issues that no other team has experienced are slim to none.

Don't get discouraged if you feel like your testing efforts are not working as planned. As the articles in this series hopefully demonstrated, there's always a way to get through these issues. Every challenge has its solution, so don't despair.

Work through the issues and reach out for help when you need it. Any good organization has team members who are willing to help with solutions. There are plenty of testers out there, including myself, who are more than happy to lend a helping hand. You don't have to go at it alone.

Have you had times where automated testing has not worked well for you? Is it something that wasn't covered in this series or the tweet by the Ministry of Testing? Let me know in the comments below, and let's figure out the issues together.

Photo credit: Bruce Hong on Unsplash