When you're beginning to write end-to-end tests for an application, a common question arises: Should I keep these tests next to the application code, or separate it into a different repo?

This question pops up over and over again online. Both experienced and new testers alike always wonder what's the best thing to do. It's a fair question to ask when the application you want to test is under a single code repository.

Many test frameworks give you the freedom to choose how you want to structure your tests. For the most part, you're not tied down to a defined standard. Unless you're using an application framework that has an opinionated way of doing things (like Ruby on Rails or Laravel), it's up to you to choose.

Which choice should you make? Let's take a look at both options.

Placing your end-to-end tests in a separate repo

Putting your end-to-end tests separately from your application means you'll have at least two code repositories. For instance, your primary application is in a repo called awesome-app, and your end-to-end tests live in another repo called awesome-app-tests.

In this scenario, each repo will contain a workflow according to its needs. The application repo can have specific tasks for building and running the application. On the other side, the end-to-end testing repo can contain jobs to set up the application under test and execute the tests.

This setup doesn't necessarily mean that the testing repo will only contain tests; the application repo can also provide its own set of tests. For instance, the application developers can write and maintain smaller unit and integration tests. These tests run alongside the application, often using built-in tools included in the application's framework.

Having a separate repo has its set up benefits. Overall, it keeps the overall development and testing environment cleaner. Each repo will contain only the things it's concerned about, so it's easier to navigate and focus on specific tasks.

In some cases, a separate repo will become more of a necessity. For example, if the test framework requires a different way to set up, it's better to separate it from the application for simplicity. In some cases, the tests need an entirely different environment to run the tests. Having the tests elsewhere facilitates this process.

There are downsides to this approach, however. The main one is that you're adding more moving parts to your workflow, which makes things more complicated along the way. You'll have to add more steps to the team's initial setup, you need to keep the application and its tests synchronized, and more.

Having a separate repo also reduces collaboration between team members. If your organization has a clear distinction between development work and testing work, a separate repo will widen that gap. Everyone may start with good intentions, but eventually, developers will focus on development, and testers will focus on testing. There will be little - if any - teamwork to keep tests running smoothly.

Another thing to look out for is for disruptions to the team's existing workflows. Having to execute commands in different places increases friction in all aspects. For example, it's annoying to forget that you need to perform a different set of commands to run the end-to-end tests in another directory. Or you need to jump through hoops to get your continuous integration system to run regression tests as part of the pipeline.

Placing your end-to-end tests in the same repo

Placing your end-to-end tests in the same repo keeps everything contained in a single place. Usually, this means a subdirectory created placed in your application's codebase, like /tests or /end_to_end.

Here, all tests become part of the development workflow. Keeping everything together means that you'll have the ability to run both application-related tasks (build, unit testing, etc.) and end-to-end tests in the same spot.

Keeping the end-to-end tests alongside the rest of your application avoids most of the issues with having them separately. The friction to the workflow that's introduced with multiple repos disappears. Developers and testers can run everything in the same codebase, keeping testing and development efforts close to each other. It also helps with the initial setup, since there's only one repo to deal manage

More importantly, you will increase the sense of ownership throughout the team. There's no barrier between developers and testers because the tests are within reach of everyone. When the tests fail, it's no longer the responsibility of only the testers. There's a mental shift when things are together. No one wants to see their builds breaking if it's in their main repo.

Still, it doesn't mean that keeping your tests and application together solves everything. There are a few scenarios where you should consider splitting them up.

One of the main reasons for having a separate repo for your tests is if the test framework needs a different toolset to function properly. You can technically have multiple frameworks and libraries living together, but it will inevitably complicate things for everyone. Developers and testers have to set up and use two different sets of utilities, and they need to know them well. You need to make sure any continuous integration systems support both toolsets. The list goes on.

Another reason to separate codebases is if you need to restrict who has access to the code. For instance, if you hire a third party to work on test automation, you wouldn't necessarily want them to have access to the application code. You can't restrict them to a sub-directory of a single repo, so you eventually have to separate their code from the rest of the application.

My experience with both approaches

Earlier this year, the company I work for decided to improve our test automation process. The company works on multiple projects at any given time. These applications ranged from being in a pre-release state to mature production projects under development for years.

Since these projects were still under active development, we made the decision early on to separate our automated test cases in different repositories per project. The main reason was to minimize the disruption the testers would have on the rest of the team. The development team could continue with their roadmap while the testing team could begin putting our new test framework to use.

The split repos worked well initially. Developers could continue doing their work and hitting their deadlines. Testers were free to learn and explore new tools while writing test cases. There was little disruption while our improvements to test automation took place.

Once the initial test suites were ready to go, we began to roll them out into the project's workflow. That's when we started to see some cracks in our strategy.

The main problem was that we accidentally created a solid barrier between the work done in the app's repo and the work done in the tests repo. Ownership split up for each type of work. People working on the application paid little to no attention to the tests, and the testers rarely ventured out of their repo.

Since we essentially created two teams working in nearly-complete isolation, our builds never became as stable as we wanted. Automated tests broke all the time because of the lack of communication between app developers and automation testers. Developers and testers weren't really up to date with what others were doing. The efficiency on both sides was negatively affected by this back and forth discussion.

After a while, we began to feel like this was not going to work in the long term. We needed to change, or else our automation efforts would be pushed aside and forgotten as a failed experiment. That's not what I wanted.

Thankfully, we had an opportunity to correct our course. Part of the organization's desire was the unite developers and testers as one team with the matching skills. Seeing that this was happening, we shifted our focus and decided to move the tests into the same repo as the app's code.

We were able to make this transition because our test framework had a familiar toolset and programming language than the application's code. But it wasn't a walk in the park. This extra work had its share of issues along the way, primarily having to delay writing new test cases and causing some interruptions to the application developers.

However, we saw improvements almost immediately that made the delays worth it.

For one, we observed that the project builds had fewer failures from the get-go. This improvement happened because the team was working on the same bits of code. There was a much higher degree of comprehension on what was going on, so the team caught issues swiftly.

Improvements helped the reduction in failed tests in our workflow. It was easier for anyone to execute test cases confidently since it was all in the same place. There were no additional toolsets that the team needed to install or learn. Our pipeline of committing code and triggering builds on our continuous integration system became more streamlined.

More importantly, a shared sense of testing ownership developed across the entire project team. Testers were paying more attention to pull requests, noting areas where changes introduced higher risk or break existing tests. Developers chipped in with fixing broken tests and even creating new tests for features. When the repos were separate, this never happened.

Summary: What should you choose for your project?

In my story above, it seems like choosing a single repo is a clear-cut winner. But just because that's what worked best for my organization, it doesn't mean it will also work best for your organization.

In my case, we were fortunate that our test framework shared standard tools with the application's codebase. We also planned to train our testers to do more coding and software development, making the transition from multiple repos to one repo easier. Your situation will vary. You might be in an entirely different boat. You might have an outsourced testing team, or need to use completely different development environments for your applications and tests.

You need to see the whole picture before you make a decision. However, there are a few rules of thumb you can follow to help you along the way.

Choose a single repo for your tests if:

• Your test framework doesn't need a different setup as the application under test.
• Your test framework shares a common programming language with the application.
• You want to use a common toolset for your team's workflow.

Choose a separate repo for your tests if:

• Your application and test framework need different environments to set up and run.
• Your test framework uses a different programming language or toolset than the application.
• You need to limit access to your application's codebase to third parties.

The following flowchart will help you make a decision on where to place your end-to-end tests, based on the guidelines above. You can also download a PDF version of this flowchart if you wish to print or share it with your colleagues.

Does your organization keep its automated tests in a single repo or a separate repo? How has either approach worked for you? Share your experiences with others in the comments below.

Photo credit: Tania Melnyczuk on Unsplash