Scaling Terraform Modules

Terraform modules let you write a piece of infrastructure code once and reuse it across your workspaces. You package the resources into a self-contained component, then call that component wherever you need it. As your deployments get more complex, that packaging is what keeps the code manageable.

Writing a module is the easy part. There's plenty of help out there, whether you build one from scratch, pull it from the public Terraform module registry, or use a provider like Hestio to manage them for you. This post skips module authoring and covers what happens when you try to scale module usage across an organization.

What Challenges Come Up When You Scale Terraform Module Usage?

When you scale your module usage, there are a few things to think through:

• How will you distribute the module to your developers?
• Once it's distributed, do the developers know which versions to use?
• How will you make sure the Terraform modules created for specific teams or applications are only available to that team?
• As an admin, do you know if everyone is pulling the correct Terraform modules and from the approved sources?
• Do you know when a module isn't being used and can be deprecated?
• Do you have the resources to support Terraform modules for your organization to avoid becoming a bottleneck?

The more modules you have and the more developers using them, the bigger this gets. Users keep asking for more modules and more services, and without a system in place it's hard to track what you have and who's pulling what.

How Does a Private Terraform Module Registry Solve These Problems?

If those challenges sound familiar, a private module registry from a Terraform Automation & Collaboration (TACO) platform like Scalr or Terraform Cloud will help. A private registry lets you curate a list of modules and share them with your organization. Most private Terraform registries give you the same core functionality:

• Create a list of available modules with instructions on how to use them.
• Versioning of the modules that are pulled from the VCS providers that they are stored in.
• Ability to copy the module code and plug it into your workflow.

Screenshot of the AWS S3 Terraform Module

That baseline is common across most Terraform module registries. In larger organizations, though, you run into scaling issues the baseline doesn't cover: RBAC, reporting, and how modules get shared between teams.

What Sets Scalr's Module Registry Apart?

Scalr's module registry differs from that baseline in four ways:

1. Namespace-based sharing. Modules live in namespaces created at the account scope, and each namespace can be shared with the entire account or a select set of environments. You can share modules broadly across your organization and still limit who has access to sensitive modules.
2. No Code Deployments. Not everyone in your organization will be up to speed on Terraform, which is why Scalr has created the ability to deploy workspaces directly from the module registry without having to write Terraform code.
3. Reporting. Once module distribution is solved, you need to know how your Terraform modules are being used. The Terraform reporting feature shows you.
4. Supported Modules. With the registry in place, the next question is whether you can maintain the modules long term or if you'll become a bottleneck for development teams. If it looks like you're heading towards becoming a bottleneck, we partner with Hestio who creates and maintains low code "patterns" through their WorX offering.

Namespace-Based Sharing

Scalr's module registry uses a namespace-based model to spare administrators a lot of operational pain. You don't want teams stepping on each other when they deploy, so Scalr lets you create environments that give each team or app its own space for their workspaces. Importing modules into every single environment by hand would be a mess. Namespace sharing is the way around that:

Illustration of the Sample Structure in Scalr

All module namespaces are created at the account scope, and for each one you choose how it's shared:

• Share with the entire account: every environment can use modules published to the namespace.
• Share with a select set of environments: only workspaces in the chosen environments can view and use the modules.

You create and maintain a module in one place and share it with many organizations, teams, and environments. That lets you set organizational standards and still scope sensitive modules to the teams and environments that should see them.

No Code Deployments

Terraform code can be deployed into workspaces by using the Terraform CLI, through PR automation with a VCS provider, or through the Scalr private module registry referred to as "no-code deployments". With this method, your users who are not as familiar with Terraform can create a workspace directly from the module registry and will be prompted to fill in any required inputs that do not already have a value. The experience is much simpler, and users still see the core Terraform workflow in action while the resources are being created.

Creating a workspace in Scalr

Reporting

Once you have figured out how to distribute the Terraform module code and have your users deploy it in their workspaces, you'll want to understand the overall usage. Telling your users to use the module registry doesn't mean they do. The Scalr Terraform reports feature gives you insights into the following:

• The number of workspaces that are using a module.
• The versions of the module that are being used.
• The source of each module.

The source matters most, because it shows whether anyone is working around controls to pull Terraform modules from a different module registry or even from a Github repository that is not approved. The reports cover compliance as well as usage.

Modules-in-use Report

Supported Modules

You also need to make sure you aren't a bottleneck for your development teams. Hestio, a Scalr partner, created low code modules that novice and advanced Terraform users alike can deploy. The modules can be shared with your organization and Hestio will take care of the maintenance for you.

Example of Hestio Module

How Should You Plan Your Own Module Strategy?

The Scalr module registry lets you manage Terraform modules safely at any size. When you sit down to plan your own module strategy, these are the areas worth thinking through:

• Pick a distribution strategy for your Terraform modules that accounts for RBAC and tenancy.
• Understand your users and the best way for them to deploy the modules.
• Track module usage so you know which modules are deployed and where they come from.
• Decide whether your organization can support Terraform modules for the long term, or whether you'll need a firm like Hestio to help.

Everything listed here is included in Scalr's free tier.