When a dev team releases code that could impact over 6 million users, it’s vital that their staging environment is conducive to accurate and reliable deployment tests.
This is the case for devs at Alignable, a Boston-based referral network for small businesses. With such high stakes, Senior Software Engineer Jerry Cavanaugh said his team has a number of processes in place to ensure their staging environments are dependable. For one, each team member can tap into AWS to build a small but true-to-live staging environment of their own, whenever they want. It allows devs to quickly test the features they’re working on and keep production moving. To keep staging environments accurate, they’re updated with clean Docker images daily, and new production data weekly.
“We grant our entire engineering team the ability to spin up complete environments at will using cloud formation templates that closely match production,” Cavanaugh said.
Without the freedom to test code, builds and updates quickly and accurately, Cavanaugh said dev teams and users could experience bugs and performance issues.
Below, Cavanaugh shared more insight into his team’s staging structure and what other devs can do to optimize their own environments.
What’s a critical best practice your team follows when developing staging environments, and why?
It’s critical that staging environments are fast to deploy, accessible and easily available to developers. If they aren’t, they become blockers to developer velocity and devs will fall back to only testing in their local environment, leading to an increase in production bugs and a decrease in stakeholder input and overall quality. We take advantage of AWS’s scalability to boot up an unlimited number of staging environments on demand and support quick, incremental deploys once an environment is up.
We grant our entire engineering team the ability to spin up complete environments at will using cloud formation templates that closely match production. These environments closely match production in configuration, although they’re typically smaller instance sizes. The settings allow developers to test not only their changes but also the deployment of those changes as well.
What processes does your team have for maintaining your staging environment?
Every night, we decommission all staging environments and full-image rebuilds from clean Docker images, and pair that with a weekly rescrub of a subset of production data. It keeps the images small, clean and replicable. And more importantly, it keeps the images up to date with the production environment.
Every night, we decommission all staging environments and full-image rebuilds from clean Docker images.”
We are constantly pushing changes to production, which means our data is constantly changing too. A weekly refresh of production data — scrubbed of any sensitive user info — on dev environments helps our engineers test things as they really are.
What’s a common mistake engineering teams make when it comes to staging environments, and what should they do to avoid it?
The most common mistake is to allow drift between the production and staging environments. This limits the ability of staging to mimic production and you’ll end up not surfacing the same issues as production. To combat this, make sure at least one of your staging environments is essentially production in everything but name. It should include everything from build processes to deployment and redeployment so that it can serve as a canary before there is any user impact.
Devs can push their branch to spin up test environments. When they’re ready for production, they’ll merge to master and deploy to a final staging environment that follows all the same build steps as production, including a full (scrubbed) database. This process ensures that any problems not caught in testing will almost certainly surface in staging.
 (1).jpeg)
