As a developer, I've lost track of how often I've had to wait around for continuous integration in the projects I work on. The usual process begins when I make some modifications on my local branch and push the code to the repository to trigger the build. I know that it's going to take at least ten minutes for the build to finish and tell me if things are good to go with my changes.
Only ten minutes. It doesn't seem like a lot, but it's enough time for me to catch up on Slack or maybe check up on unread work emails. Other times, I guiltily pick up my phone and mindlessly scroll through my social media feeds, letting the time melt away. When the CI service finally shows me a green result, I can move on. But by this time, I get so distracted with other things that I forget what I was going to do next. Even worse, if there's an issue with the build, getting back to what I was doing a mere ten minutes earlier seems like a massive chore.
This scenario doesn't happen occasionally. It happens multiple times a week, sometimes dozens of times in a single day. It's time that employees never track, nor time that freelancers add to their timesheet for their clients. The time we spend waiting around for CI builds to complete is something that we learned to live with as part of our workdays. After all, a few minutes here and there doesn't harm anybody, right?
Maybe ten minutes isn't that big of an issue for you and your team. But the average CI run takes much longer than that. A 2022 survey by Incredibuild found that it takes 20 minutes for the average CI build to complete, with 24% of respondents saying it takes over 30 minutes. These longer build times have a massive impact on the development cycle and productivity. Yet we rarely take a step back to ask ourselves the question: what are those build times really costing you?
The Obvious Cost: Wasted Time
For most, the very first thing that comes to mind when asking this question is time. It's one of the more tangible effects of a slow CI build, since it's something we can easily track through a dashboard. Most CI services track these details over time and provide analytics through their reporting systems. Build time feels measurable, which makes it feel manageable and something we can easily optimize.
Let's do some simple "back of the napkin" math using Incredibuild's average of 20 minutes from their survey to see how much time is actually used up waiting for a CI build to finish. A developer triggering 5 CI builds per day means 100 minutes of CI waiting time. That equals over eight hours per week—a full workday. Multiply that number by the number of developers on a team, and the time spent dealing with slow CI builds stacks up rapidly across the board.
Of course, these numbers are only assumptions, and we're not sitting around doing nothing during this time. We often use that time productively, like reviewing our teammate's pull requests or opening a new one for the work we just pushed. We can reply to questions we deferred while we were heads-down in our work. Most of us have countless small things we can handle during these 10-20 minute waits.
However, wasted time only accounts for a small fraction of the cost we pay for slow CI builds.
The Hidden Cost of Slow CI Builds
Trying to optimize the time we spend waiting around for CI builds frequently masks a more profound problem. When we're waiting for a CI run to wrap up, we jump onto something else. Some examples mentioned earlier are reviewing code, checking unread Slack messages, replying to work emails, and so on. These feel like good uses of our time, right? Unfortunately, these actions cause context switching. For developers and testers, they can be the number one killer of productivity, since having to switch tasks comes at a cost.
If you're constantly being pulled in multiple directions at work, you know how much this affects your productivity throughout the day. Many studies around context switching have uncovered how much a single interruption impacts us, like one from the University of California, Irvine, that uncovered it takes an average of 23 minutes and 15 seconds to return to a task. We don't consciously think about this, but if you were to track how long you need to get fully back to what you were doing after an interruption, you'd discover this study is spot-on.
Using the results of the UC Irvine study, let's return to the hypothetical calculations made earlier, where a developer triggers about five CI builds in an average day. While waiting for those CI runs to finish, the developer switches their brain to focus on other work or chatting with co-workers. Twenty minutes later, the CI run wraps up, and the developer switches back to what they were doing. That context switch is going to eat at least 15 minutes of productive time. Between the CI build time and the context switching time, you're now looking at over 14 hours of lost focus in a given week. Again, these are hypothetical numbers, but they can easily reflect your reality.
When teams begin noticing the hidden dangers of the constant context switching that happens due to slow CI builds, it begins to shift the team's behavior. But it's not always a positive shift.
The Cost Around You and Your Team's Behavior
Once we know how context switching wrecks our productivity, our behavior changes in an attempt to mitigate our lost focus. To deal with slow builds, the team's workflow begins to move down the road of minimizing the times they trigger their CI service. It often happens unconsciously and without the team realizing they're glossing over the problem instead of fixing the root cause. Ironically, trying to circumvent a slow CI build makes the development workflow slower over time.
One way that developers try to sidestep CI is by pushing fewer changes that trigger a CI build. Usually, this means the changesets are much larger in scope. For example, instead of incremental code changes consisting of 2-3 commits, a developer might wait days to finish a feature before pushing everything at once. While it only triggers one CI build instead of multiple builds, doing so increases the surface area of what's checked. It's a lot more difficult to debug and address an issue hidden in dozens of commits compared to a handful of smaller changes. Triggering fewer CI runs is typically accompanied by the lack of adding new tests to avoid slowing down the builds even further, further reducing the quality of the codebase.
Another typical change around these behaviors is an erosion of trust in the CI process. It starts with fewer test runs, followed by less maintenance around the test suite, which leads to no one wanting to wait around for the builds to finish and merge their changes without verifying them. Before they know it, it's the Wild West in the codebase, and the team is left wondering why they have to deal with so many issues in production. It's a vicious cycle that I've seen time and time again drastically slow down a project.
It always starts with one person making fewer changes and trying to figure out how to avoid the real problem of their slow continuous integration processes. Unfortunately, these bad habits can become contagious, although not on purpose.
The Team Cost
Most developers don't change their working patterns intentionally when it comes to slow CI builds. What usually happens is a trickle-down effect where one person's work causes the entire system to slow down and block everyone else from moving forward with their tasks. Depending on a team's workflow, a slow CI build can become a bottleneck that creates a pile-up of pending tasks that can only proceed as quickly as the CI run lets them.
Let's take an example I've seen happen before in a project where the CI run took about 20 minutes. One developer would open a pull request with a rather large batch of work since they wanted to minimize the time they had to wait around for the build to finish. Because they weren't pushing new changes frequently, the CI build often flagged issues in their changes, and the developer had to context-switch back into a branch they thought was done. This slow feedback loop caused them to take much longer to fix their branch, so that pull request was blocked until they could get CI to pass.
Meanwhile, other developers on the team are pushing their code updates. Since the CI service limited the number of concurrent builds, theirs would get queued without knowing how long it would take to reach their turn. A full code review couldn't take place because the CI run hadn't happened, so pull requests piled up. These blocked PRs would also create a higher possibility of merge conflicts between branches, leading to even more work. During periods of high activity—usually just before a major release—it would take days to merge some pull requests. No one wants to wait that long for their work to be reviewed, let alone before a big production deployment.
Slow CI builds aren't an isolated event. They create problems that pass through the entire team and negatively impact everyone working on the project. If we're not careful, it can trigger even worse results across the organization.
The Immeasurable Cost
Up to this point, we've looked at the more tangible effects that slow CI builds have on a development team. We can track build times, attempt to mitigate the costs of constant context switching, and check how long our pull request queues become. But there's another layer of damage that doesn't show up in any dashboard or report, and it might be the most significant cost of all. The daily grind of waiting for slow CI builds wears people down in ways that are difficult to quantify but impossible to ignore.
When developers start to feel that CI is making them not produce their best work, they'll begin to hate it. That resentment burns up rather quickly. First, they resent having to wait for a test suite that takes so long to run. Before long, they resent the codebase itself, and eventually, they start resenting their job. I've watched talented people completely disconnect mentally from projects because every small change felt like pushing a boulder uphill, and admittedly, I've experienced it myself. The enthusiasm that once drove them to give it their all fades into a dull acceptance of how things are. Speaking from personal experience, this shift rarely fixes itself. I've called it my "point of no return."
Once this happens, the quality of the work and the team's morale take a beating. When every change takes twenty minutes or more to validate, developers stop taking chances. They avoid refactoring messy code because the risk of breaking something isn't worth the wait to find out. They skip writing that extra test because it would add another few seconds to an already bloated system. They push code they're not entirely confident about because they just want to move on. Slow CI builds feel like wading through sludge, and eventually, people stop trying to move at all.
This disintegration of morale and quality doesn't appear on any organization's bottom line, but its effects compound over time.
Why Teams Don't Fix Their Slow CI Builds
If slow CI builds cause so much damage, why do teams let them linger around for far too long? The most common answer I hear is some variation of "it's not that bad." The normalization of poor CI processes happens gradually. A build that once took five minutes creeps up to ten, then fifteen, then twenty. Because the change is incremental, no single day feels dramatically worse than the one before it. The team adjusts their expectations and their workflow without ever making a conscious decision to accept the slowdown. Before long, a twenty-minute build feels normal because everyone has forgotten when it only took five minutes.
Another problem is that CI often falls into an ownership void. Most developers see it as infrastructure, so it's not their responsibility. QA teams typically don't maintain CI services. DevOps might have set it up initially, but often they've moved on to other priorities in the project. That leaves no single person or team to feel accountable for the CI pipeline, so improvements happen sporadically. On top of that, the lack of ownership makes it tough to justify spending time on it when there's a backlog of features waiting to ship. Optimizing CI doesn't produce a visible deliverable that stakeholders can point to, so it's easy to put this work on the back burner.
Perhaps the most deceptive reason teams don't fix their slow CI builds is that the cost to the organization is nearly invisible at a grander scale. Since the pain is spread across the entire team, it rarely reaches that breaking point that spurs someone into immediate action. As mentioned many times throughout this article, it's difficult to measure how much this affects the work and the team. So the organization ends up under-investing in their CI process, and the project ends up dying a slow death by a thousand cuts.
Even if you feel like your team has reached this point, the good news is that there is hope. Understanding why teams tolerate slow CI builds is the first step toward changing the pattern.
Wrap Up: What You Can Do About Your Slow CI Builds?
The goal of this article isn't to hand you a prescriptive list of technical solutions. Every codebase and organization is different and has its set of unique challenges. The specific changes that will make the biggest impact depend on factors that fit your situation. Instead, here are a few starting points that can help you and your team begin to rein in your slow CI builds.
The first step I often take is to start by measuring. It sounds obvious, but many teams have no idea what their average build time actually is, let alone whether it's trending upward or downward over time. Most CI services provide this data, but few teams review it regularly or have even looked for it. Once you have a concrete number, you can start having conversations about whether that number is acceptable and what it would take to improve it.
From there, the next step is to assign ownership. Someone on the team should care about the CI pipeline and feel responsible for keeping it in check. It doesn't have to be their full-time job, or it can come from outside help, but it does have to be someone's role. The most successful teams I've worked with, as far as keeping their code quality high, are those that treat CI like a product with someone in charge. It's not something that's merely a small part of their workflow that's only noticed when there's a problem. If your CI pipeline only receives attention when it breaks, you're guaranteed to fall behind.
Continuous integration processes are essential for modern software development, but they do come at a cost. It's our job to make sure that we keep those costs to a minimum to create a return on that investment. Think of it this way: your build time is a tax on every change your team makes. Every feature, every bug fix, every experiment, and every refactor pays that tax before it can ship. What's your tax rate, and who's tracking it?
