Scaling Terraform Modules

Terraform modules are a great way to simplify your Terraform code by writing it once and then reusing the modules across your Terraform workspaces. Terraform modules enable you to encapsulate and organize your infrastructure code into smaller, modular components. This modular approach helps in managing and scaling complex infrastructure deployments.

The creation of modules is relatively simple and there are a ton of resources to help you get started whether you write the module from scratch, pull it from the public Terraform module registry, or use a provider like Hestio to manage them for you. In this blog, we aren’t going to talk about creating the modules, but rather about scaling the usage of Terraform modules.

Terraform Module Scalability

When scaling your module usage, there are a number of factors to consider:

• How will you distribute the module to your developers?
• Once it is distributed, do the developers know which versions to use?
• How will you make sure the Terraform modules created for specific teams or applications are only available to that team?
• As an admin, do you know if everyone is pulling the correct Terraform modules and from the approved sources?
• Do you know when a module is not being used and can be deprecated?
• Do you have the resources to support Terraform modules for your organization to avoid becoming a bottleneck?

The more Terraform modules you use and the more developers you have, the bigger the problem becomes. Users will always push for more modules and more services, but in the current state, it will be hard to keep track of what Terraform modules you have and who is using it.

Private Terraform Module Registry to the Rescue

If these challenges apply to you then you would likely benefit from a private module registry from a Terraform Automation & Collaboration (TACO) platform like Scalr or Terraform Cloud. Terraform private module registries allow you to create a curated list of modules to share with your organization. You’ll find the following common functionality across most private Terraform registries:

• Create a list of available modules with instructions on how to use it.
• Versioning of the modules that are pulled from the VCS providers that they are stored in. 
• Ability to copy the module code and plug it into your workflow.

As mentioned, the list above is common across most of the Terraform module registries, but you’ll still have scaling issues within larger organizations around things like RBAC, reporting, and sharing of the modules.

Scalr Differentiators

Scalr has a few key differentiators in the way the module registry has been implemented:

1. Inheritance - Modules can be created in a registry at the account or environment level, which then makes it easier to share the modules across your organization and also limit who has access to certain modules.
2. No Code Deployments - Not everyone in your organization is going to be up to speed on Terraform, which is why Scalr has created the ability to deploy workspaces directly from the module registry without having to write Terraform code.
3. Reporting - Once you have figured out the module distribution problem, you’ll then need to figure out the usage of your Terraform modules. The Terraform reporting feature gives you insights into the module usage.
4. Supported Modules - Now that you have implemented the module registry strategy, you’ll need to figure out if you can maintain the modules long term or if you’ll become a bottleneck for development teams. If it looks like you’re heading towards becoming a bottleneck, we partner with Hestio who creates and maintains low code “patterns” through their WorX offering.

Hierarchical Inheritance

Scalr created the module registry with hierarchical inheritance in mind to save administrators from an operational nightmare. The last thing you want is for teams to step on each other's feet when deploying, in Scalr you can create environments to give each team or app its own dedicated space to manage their deployments or workspaces. It would be an operational mess to have to go into every single environment and import the modules, which is where the inheritance model comes into play:

‍

A module registry can be created at each level in the diagram above:

• Account scope: If a module is created at the account scope, all environments within the organization inherit the module.
• Environment scope: If a module is created at the environment scope, only workspaces in that environment can view and use the module.

Create and maintain the module in one place and share with many organizations, teams, and environments. This allows you to create organizational standards, but also lets your individual teams add more modules at their respective scope if their permissions allow for it, but not change or remove modules from a higher scope.

No Code Deployments

Terraform code can be deployed into workspaces by using the Terraform CLI, through PR automation with a VCS provider, or through the Scalr private module registry referred to as “no-code deployments”. With this method, your users who are not as familiar with Terraform can create a workspace directly from the module registry and will be prompted to fill in any required inputs that do not already have a value. This greatly simplifies the overall experience, but the users will still see the core Terraform workflow in action while the resources are being created.

‍

Reporting

Once you have figured out how to distribute the Terraform module code and have your users deploy it in their workspaces, you’ll want to understand the overall usage. Just because you have told your users to use the module registry doesn't mean they are doing that. The Scalr Terraform reports feature gives you insights into the following:

• The number of workspaces are using a module.
• The versions of the module are being used.
• The source of each module.

That last point is critical as you’ll be able to identify if anyone is working around controls to pull Terraform modules from a different module registry or even from a Github repository that is not approved. The reports not only help you understand the module usage, but also ensure compliance.

‍

Supported Modules

As mentioned, you’ll need to make sure you are not a bottleneck for your development teams. Hestio, a Scalr partner, created low code modules that the most advanced or novice Terraform users can deploy. The modules can be shared with your organization and Hestio will take care of the maintenance for you.

Summary

With the Scalr module registry, you can easily and safely manage Terraform modules for any size organization. The key areas to think about when creating your module strategy is to:

• Figure out the distribution strategy for the Terraform modules and one that has the proper RBAC and tenancy in mind.
• Understand your users and the best method for deploying the modules.
• Module usage to show modules are being deployed and from where.
• Lastly, is your organization well equipped to support Terraform modules for the long term or will you need to work with a firm like Hestio to ensure long term success.

Try it out in Scalr today, everything listed here is included in Scalr’s free tier.