It's Friday morning at the office. You and your team have been working hard to wrap up the latest sprint, and it's just about ready to go. The QA team has put the app through its paces. After one last run-through, there aren't showstopping bugs holding the team back. The product team has been waiting a long time to push out some new features and patch those annoying bugs. They can't wait to see it live for the world to see. All that's needed is to start the deployment process, and the celebration can begin.
However, no one is willing to begin the process. Everyone seems hesitant even to bring up the topic, hoping that the next couple of hours pass by quickly so everyone can enjoy their weekend. Finally, a new member of the team asks the question no one wanted to hear:
"... so... should we deploy to production?"
In most organizations, asking this question on a Friday will cause evil glares and other team members to question your sanity. If you've worked in software development, chances are you've heard the phrase "We never, ever deploy on Friday". It's repeated so often that it's a widespread meme at this point:
I fully understand this sentiment. No one wants to risk interrupting their free time on Saturday or Sunday by having to jump into work and fix a bad deployment. Unless it's a highly urgent bug fix, any code deployment can wait a few days so the team can come in refreshed from their weekend and with more time available to handle any potential issues.
However, I firmly believe the for God's sake, don't deploy on Friday attitude on most development and testing teams is the wrong one to foster in an organization if you want to build a high-quality application.
Why avoiding Friday deploys is an unhealthy habit for quality
When a team avoids Friday deployments like the plague, it usually boils down to a lack of confidence in their application, processes, or both. The reluctance typically stems from the organization needing to know or trust that their applications are ready to let its customers use them. Any time someone says "We won't deploy on Friday", what they're really saying is "We don't trust our application or processes to deploy on Friday."
In the long run, that thought doesn't lead to a high-quality system. Teams don't often use these feelings of uncertainty as a signal that they need to focus on improvement. Whether the lack of trust and confidence is a real issue or an imagined one, most teams take the easy route by glossing over the problems and opting instead to mask them. Rather than spend time finding improvements, they come up with rules like no-deploy Fridays.
Instead of avoiding deployments towards the end of the week, how about tackling the problems head-on and enhancing existing processes to give teams the freedom to deploy any time they need? As a developer or tester, how would it feel to not worry about having your weekends interrupted because you had to update your application just before punching out of work?
Any organization serious about driving quality across their teams and products must have that trust built into their company. While it takes time to feel completely confident in your systems, the good news is that it's doable for anyone, no matter how messed up they think their application and processes are in their organization. The following three-step plan can get you there faster than you think.
Step 1: Implement a stable automated test suite
An automated test suite is a must for any software development team that wants to build and ship quickly without constantly worrying about their application. You're severely undermining your team's ability to work and deploy confidently without one. If you have yet to give yourself or your team the space to work on expanding automated test coverage, you're purposely putting yourself at a disadvantage.
The first step for any organization to improve its quality levels is to focus on having enough test coverage on the critical paths of your application, so focus on that step if you're starting from scratch. However, something even more important than coverage is stability. Even with full coverage, your test automation won't be helpful if it constantly fails. If you have flaky tests, you won't ever feel comfortable deploying at any time—let alone on Fridays. While working on your test coverage, focus on building a robust, maintainable test suite to keep your team's confidence high for the long term.
Another area of focus is the type of testing you should do. Depending on your application's needs, you'll likely need a mix of various automated tests. Some teams write a few unit tests and call it a day, but today's modern applications have lots of moving parts you'll need to check. End-to-end tests, validating performance, and ensuring security and accessibility are just the tip of the iceberg—plan which types of automated tests can benefit your organization ahead of time.
Know what kinds of tests you need and the areas you want to cover. Having a stable automated test suite with the right mix of tests will reinforce your team's confidence in your applications and deployments.
Step 2: Don't forget the manual exploratory testing
When teams get into the habit of building stable automated tests for their applications, they might become overconfident in their test suite's ability to catch issues before deployment. Often, a false sense of security leads people within the organization to think that they don't need to perform additional testing activities with their test automation practices. This train of thought can't be any farther from the truth. No matter how good your test automation habits are or how much coverage your written tests provide, you can't ignore the importance of manual, exploratory testing.
Automated tests can only do what they're told to do, serving well to catch regressions during rapid development cycles. On the other hand, manual and exploratory testing will allow you to look into the blind spots of your automated tests and discover problems in areas no one thought to look at—the so-called "unknown unknowns". Neglecting this essential part of testing while relying on automation inevitably leads to a lower-quality product.
Manual testing takes time and effort to perform well. Some teams, especially smaller ones without dedicated QA teams, tend to perform these tests at the last minute. Most startups I work with only do exploratory testing a few days or even hours before a significant deployment. Sometimes it works, but you risk rushing through the process and letting issues slip by.
Ideally, manual exploratory testing should happen consistently throughout the team's development cycles as new builds get pushed out to a staging environment. Give your organization the time to do this testing, and you'll find yourself worrying less about deploying at any time.
Step 3: Have an automated build and deploy process in place
One part of the development and release cycle often forgotten in organizations building software is the build and deploy process. In some places I've worked, building and deploying a new release feels almost like a mystical process that only a few chosen ones can perform. Their deployments usually consist of a long process of steps performed in a precise order with almost-perfect timing. If the person performing this ritualistic act slips along the way, the fragility of the process can bring down the entire application in an instant.
The way I describe this process might be slightly exaggerated, but it's pretty close to the truth. Many companies I've worked with have unnecessarily complex deployment procedures. When asked why, some teams try to justify why it needs to be the way it is, but the subtext behind their response really means "We've always done it this way, so we never changed it." The common reason is that someone long ago created this complicated process. Since it's working—no matter how delicate or convoluted the process has become—they never bothered to improve the status quo.
If your team has to perform multiple steps to build and deploy a new version of your app, you're doing a massive disservice to your organization. No matter how many moving parts your entire system contains, you can automate almost any release process—even condensing it down to a single command. We have tons of excellent tools nowadays that make it dead simple to automate the build and deployment process, from Jenkins to AWS CodePipeline to CircleCI and countless others. Your organization has no excuse to avoid automating your deployments.
An equally important and forgotten part of the deployment process is rolling back a failed deployment. Despite a well-tested automated process, it can still fail for many different reasons, from bad code getting merged to glitches in your application's infrastructure. Most teams discover that they don't have any rollback strategies when a deployment fails and their application is down. Before this happens, develop a rollback strategy that suits your situation and test it frequently to ensure you aren't surprised when Murphy's law strikes.
Summary
Asking a software developer or tester to deploy on a Friday is usually treated as a cardinal sin. Most teams refuse to deploy anything on a Friday for fear of breaking something and needing to work over the weekend. While it's a reasonable thing to do, the sentiment typically comes out of a lack of confidence in the organization's deployment processes. In most cases, the fear of a new release breaking the system isn't limited to Friday deploys but to deploys that occur at any time. This sentiment isn't conducive to building quality in an organization.
If your team treats deployments as a fragile procedure that can bring everything down, you can take a few steps to help bolster everyone's confidence in the process:
- Have a stable automated test suite that provides the right blend of testing and coverage to catch issues before going out to production.
- Establish a consistent habit of conducting manual exploratory testing to uncover bugs your test automation can't see.
- Finally, automate your deployment steps and ensure you can revert any release. No process is perfect, and there will likely be times when you will have to roll back a deployment.
With these steps, your team will be ready to quickly resolve the problem in the rare instance something goes wrong. Once you have some or all of these suggestions in place, you'll notice the quality of your applications will increase. More importantly, it'll boost the team's confidence along the way.
The purpose of this article isn't to have your team deploy every Friday—it's to give them the option to do so if needed without any concerns. The ability to deploy without worrying that something will figuratively explode lets every developer and tester have one less thing to worry about during their project cycles. And if we're lucky, maybe we'll see fewer of those memes cluttering up our LinkedIn feeds.
