Scaling Terraform Modules

Terraform modules let you write a piece of infrastructure code once and reuse it across your workspaces. You package the resources into a self-contained component, then call that component wherever you need it. As your deployments get more complex, that packaging is what keeps the code manageable.

Writing a module is the easy part. There are plenty of resources to help, whether you build one from scratch, pull it from the public Terraform module registry, or use a provider like Hestio to manage them for you. This post isn't about creating modules. It's about what happens when you try to scale their usage across an organization.

Terraform Module Scalability

When scaling your module usage, there are a number of factors to consider:

• How will you distribute the module to your developers?
• Once it is distributed, do the developers know which versions to use?
• How will you make sure the Terraform modules created for specific teams or applications are only available to that team?
• As an admin, do you know if everyone is pulling the correct Terraform modules and from the approved sources?
• Do you know when a module is not being used and can be deprecated?
• Do you have the resources to support Terraform modules for your organization to avoid becoming a bottleneck?

The more modules you have and the more developers using them, the bigger this gets. Users keep asking for more modules and more services, and without a system in place it gets hard to track what you have and who is pulling what.

Private Terraform Module Registry to the Rescue

If those challenges sound familiar, a private module registry from a Terraform Automation & Collaboration (TACO) platform like Scalr or Terraform Cloud will help. A private registry lets you curate a list of modules and share them with your organization. Most private Terraform registries give you the same core functionality:

• Create a list of available modules with instructions on how to use them.
• Versioning of the modules that are pulled from the VCS providers that they are stored in.
• Ability to copy the module code and plug it into your workflow.

Screenshot of the AWS S3 Terraform Module

That baseline is common across most Terraform module registries. In larger organizations, though, you run into scaling issues the baseline doesn't cover: RBAC, reporting, and how modules get shared between teams.

Scalr Differentiators

Scalr has a few key differentiators in the way the module registry has been implemented:

1. Namespace-based sharing - Modules live in namespaces created at the account scope, and each namespace can be shared with the entire account or a select set of environments. This makes it easy to share modules broadly across your organization while still limiting who has access to sensitive modules.
2. No Code Deployments - Not everyone in your organization is going to be up to speed on Terraform, which is why Scalr has created the ability to deploy workspaces directly from the module registry without having to write Terraform code.
3. Reporting - Once you have figured out the module distribution problem, you'll then need to figure out the usage of your Terraform modules. The Terraform reporting feature gives you insights into the module usage.
4. Supported Modules - Now that you have implemented the module registry strategy, you'll need to figure out if you can maintain the modules long term or if you'll become a bottleneck for development teams. If it looks like you're heading towards becoming a bottleneck, we partner with Hestio who creates and maintains low code "patterns" through their WorX offering.

Namespace-Based Sharing

Scalr's module registry uses a namespace-based model to spare administrators a lot of operational pain. You don't want teams stepping on each other when they deploy, so Scalr lets you create environments that give each team or app its own space for their workspaces. Importing modules into every single environment by hand would be a mess. Namespace sharing is the way around that:

Illustration of the Sample Structure in Scalr

All module namespaces are created at the account scope, and for each one you choose how it's shared:

• Share with the entire account: every environment can use modules published to the namespace.
• Share with a select set of environments: only workspaces in the chosen environments can view and use the modules.

Create and maintain the module in one place and share with many organizations, teams, and environments. This allows you to create organizational standards while still scoping sensitive modules to the teams and environments that should see them.

No Code Deployments

Terraform code can be deployed into workspaces by using the Terraform CLI, through PR automation with a VCS provider, or through the Scalr private module registry referred to as "no-code deployments". With this method, your users who are not as familiar with Terraform can create a workspace directly from the module registry and will be prompted to fill in any required inputs that do not already have a value. This greatly simplifies the overall experience, but the users will still see the core Terraform workflow in action while the resources are being created.

Creating a workspace in Scalr

Reporting

Once you have figured out how to distribute the Terraform module code and have your users deploy it in their workspaces, you'll want to understand the overall usage. Just because you have told your users to use the module registry doesn't mean they are doing that. The Scalr Terraform reports feature gives you insights into the following:

• The number of workspaces that are using a module.
• The versions of the module that are being used.
• The source of each module.

That last point is critical as you'll be able to identify if anyone is working around controls to pull Terraform modules from a different module registry or even from a Github repository that is not approved. The reports not only help you understand the module usage, but also ensure compliance.

Modules-in-use Report

Supported Modules

As mentioned, you'll need to make sure you are not a bottleneck for your development teams. Hestio, a Scalr partner, created low code modules that the most advanced or novice Terraform users can deploy. The modules can be shared with your organization and Hestio will take care of the maintenance for you.

Example of Hestio Module

Summary

The Scalr module registry gives you a way to manage Terraform modules safely at any size. When you sit down to plan your own module strategy, these are the areas worth thinking through:

• Figure out the distribution strategy for the Terraform modules and one that has the proper RBAC and tenancy in mind.
• Understand your users and the best method for deploying the modules.
• Module usage to show modules are being deployed and from where.
• Lastly, is your organization well equipped to support Terraform modules for the long term or will you need to work with a firm like Hestio to ensure long term success.

Try it out in Scalr today, everything listed here is included in Scalr's free tier.