How To Structure Terraform Deployments At Scale

Figuring out how to structure your Terraform deployments once you've hit a certain scale is a common issue across the community.

From an administrator's perspective... Do you have a defined way of creating repositories? Do you create a list of approved modules? How do you keep up with what the teams are doing? Do you give engineers autonomy? Do you enforce policy?

From an engineer's perspective... Should I put all resources in a single monolithic deployment? How do I reuse code? Should I break the deployment up into small components or modules? Am I following standards?

There isn't one "right" way to do this, but there are plenty of "wrong" ones. It depends on your use case, and there are many ways to structure Terraform deployments at scale. What helps is a common framework that can accommodate different workflows and the requirements that will keep changing on you.

Step #1: How Do You Decouple Terraform Administration and Operations?

As you scale Terraform across your organization, you have to keep the central DevOps team from becoming a bottleneck. That means giving developers more autonomy, and it means reducing blast radius at the same time.

To make this easier, at Scalr we've built a flexible hierarchical model that fits different personas and business needs. There are three scopes in the hierarchy:

• Account Scope - Think of this as an administrative control plane for your central DevOps team.
• Environment Scope - Think of this as a landing zone, an organizational unit, or similar to a cloud provider account.
• Workspace Scope - The individual workspace where a Terraform/OpenTofu configuration is managed and runs execute, with its own variables and settings.

If you were to compare this to AWS, the Scalr account would be an AWS master account and the Scalr environments would be similar to the AWS sub accounts where infrastructure is provisioned. The hierarchy makes it very easy for you to centralize the Terraform administration while decentralizing the Terraform operations.

The Scalr hierarchy

Centralizing Terraform administration

The DevOps team, who creates and maintains Scalr environments, VCS providers, provider credentials, module registries, OPA policies, teams, and agents, mostly works at the account scope. You manage all of these objects from the account scope and assign or share them with environments as needed (note: the Scalr provider can manage these objects too). The DevOps team adds any of these from the account scope without flipping back and forth between contexts to make changes.

On top of managing objects from one place, the administrator also gets operational views across all environments and workspaces in a single dashboard. That kind of observability really pays off for large deployments, as you can see in the OPA dashboard below:

The Open Policy Agent dashboard

In the example, you can see OPA checks across all environments and workspaces from one dashboard, which saves you from going into each workspace individually.

You might also want to know which workspaces are using the most runs or taking the longest, so you can spot trends across your teams. Usage reports show you that:

Observe environment/workspace usage

Decentralizing Terraform operations

A Scalr environment is a logical grouping of Terraform workspaces that have a common set of policies, provider credentials, and teams assigned to it. We usually see environments aligned to applications, functions, or cloud accounts. A workspace, the child of an environment, is the result of a Terraform deployment. In the simplest terms, a workspace holds the Terraform state, variables, and run history.

Engineers mostly work at the environment level, where they create workspaces and run Terraform through their own workflows, whether that's the Terraform CLI or VCS driven.

The workspace list

Much like isolated cloud accounts, Scalr environments are isolated areas where teams and users can work without stepping on each other. Every environment and workspace inherits the objects assigned to it from the account scope. A good example is setting a variable, like a tag or region, at the environment scope and having all child workspaces inherit it automatically. The engineers in that environment don't have to think about the standard variables, they just focus on their deployments.

Step #1 takeaway

The Scalr hierarchy gives engineering teams isolated environments while the central DevOps team keeps control and visibility from the account structure. Now that the basics are in place, let's look at how the same structure helps you reuse your Terraform code.

Step #2: How Does Reusable Code Drive Autonomy and Standards?

People have different opinions on how to structure Terraform, but we're convinced reusable code is the easiest way to standardize modules and scale Terraform. As we said in step #1, a hierarchy helps with standardization and autonomy, and the Scalr module registry helps with the reusable code. The module registry follows the same hierarchical model we already outlined. You can place a module at the account or environment scope. Put it at the account scope and every environment inherits it automatically, so any workspace can use it. Put it at the environment scope and only workspaces in that environment can use it. This lets engineers easily find and consume approved modules. The module registry is straightforward, so the real question is: how do you set up Scalr environments to make code reuse work?

Following your cloud provider model

As we said earlier, you can structure environments many ways, but for most customers it makes sense to follow the same tenancy model you already use in your cloud providers. Most of our customers have cloud accounts or subscriptions that map to a group, an app, and so on. If that works for you in the cloud, it's perfectly fine to set up Scalr environments the same way. Breaking out environments like this reduces the blast radius of changes and lets you put the right policies on non-prod and prod environments. You can also manage and vend environments through the Scalr Terraform provider, just like you would any other cloud provider.

Next, you need to figure out what a workspace means to you. This one's more of a judgment call, but we've seen a growing trend toward more workspaces with fewer resources each, which makes for quick deployments and easy troubleshooting. The code in the workspaces points to your modules from the registry, and the only thing that changes per workspace is the variable file(s), which is what makes the code reusable.

Each app and its environment has a dedicated Scalr environment

In the example above, each group has its own Scalr environment to deploy their Terraform workspaces. Each environment inherits modules, credentials, OPA policies, and variables automatically, depending on which group it is. If the team is using the standard modules from the account scope, they can inject their own extra variables right into the workspace configuration to reuse the code. They also have complete autonomy over their environment and workspaces, which decentralizes the Terraform operations.

Variables in the workspace are automatically inherited from the account scope

Step #2 takeaway

The Scalr account and environment structure isolates your teams without adding operational overhead. And because the module registry follows the same tiered approach, the central DevOps team gets a framework to promote code reuse and give engineering teams more autonomy.

How Do You Structure Terraform Deployments at Scale?

You can automate, standardize, and structure Terraform deployments in many ways, but most of it comes down to setting up modules so they're easy to find and reuse. The Scalr hierarchy does that for organizations of any size through its inheritance model and its administrative control plane.