Getting Started with Terraform Modules

Learn about different ways of storing, naming and structuring your Terraform modules.

Brendan ThompsonAugust 19, 2022

This post is part of a series on Terraform Modules Explained.

Modules are a foundational feature of Terraform and one of the most widely used features at that, this does not mean however that when they are used their use is well thought out. During my professional time working with Terraform I have seen some pretty interesting things when it comes to Terraform, whilst most of these things are generally benign I tend to find the extreme overuse of Terraform modules to be one of the most potentially dangerous. This behavior led me to write a post in August of 2020: Terraform; to module, or not to module with the sole intent of guiding engineers on when is a good time to create a module, the most important thing I wanted from that post however was to increase thought around the process of creating modules and not just doing it for the sake of it.

In today's post we will dive into a few important areas:

Hosting options
Naming modules
File structure

Hosting Options

There are a few ways we can host our Terraform modules, in some cases we may be restricted to a particular method due to security or governance requirements.

Local Module — where the module is on the local file system and referenced using a relative or exact path. Modules set up in this manner are very difficult to version as you would be versioning the entire repository that they and any other code exists within.

module "awesome_module" {
  source = "../modules/awesome-module"
 
  ...
}

Remote Module with Git — the module is hosted in a git repository of some description, this module is then invoked by referencing the git repository. When using git references we can be explicit about what version (git tag) or branch to use for our module. There are also convenience aliases for common git providers.

// Generic non-versioned git
module "awesome_module" {
  source = "git::https://github.com/BrendanThompson/awesome-module.git"
 
  ...
}
 
// GitHub explicit version
module "awesome_github_module" {
  source = "github.com/BrendanThompson/awesome-module?ref=v1.0.0"
 
  ...
}

Using the ?ref= URL parameter at the end of our git endpoint we can specify a git tag, a branch, or even a particular commit.

HTTP URLs — where you do not want to have complex module sources then an HTTP endpoint can be used as a vanity URL for the module.

module "awesome_module" {
  source = "https://terraform.brendanthompson.com/modules/awesome-module"
 
  ...
}

The above would simply do a redirect to our GitHub (or another git provider) endpoint as per the Remote Module with Git. With the HTTP endpoint, it is also possible to utilize an endpoint that returns a zip/tar.bz2/tar.gz/tar.xz archive, like the below:

module "awesome_module" {
  source = "https://terraform.brendanthompson.com/modules/awesome-module.zip"
 
  ...
}

Alternatively, it can use the archive query parameter:

module "awesome_module" {
  source = "https://terraform.brendanthompson.com/modules/awesome-module?archive=zip"
}

Cloud Storage — finally Terraform allows the use of directly pulling modules from GCP and AWS cloud storage. It is important to note in both scenarios credentials will be required to allow Terraform to access the storage accounts.

// AWS
module "awesome_module" {
  source = "s3::https://s3.ap-southeast-2.amazonaws.com/brendanthompson/modules/awesome-module.zip"
 
  ...
}
 
// GCP
module "awesome_module" {
  source = "gcs::https://www.googleapis.com/storage/v1/brendanthompson/modules/awesome-module.zip"
 
  ...
}

Module Registry Protocol — The most powerful way to source modules (in my opinion) is through a service that implements the Module Registry Protocol (MRP), both Terraform Cloud and Scalr are examples of that. By using MRP we can declare a specific version or be more flexible with version constraints such as pessimistic versioning.

// HashiCorp Public Repository
module "awesome_module_public" {
  source = "ministry-of-magic/awesome-module/azurerm"
  version = ">= 1.0.0, < 2.0.0"
 
   ...
}
 
// Terraform Cloud
module "awesome_module_tfc" {
  source "app.terraform.io/ministry-of-magic/awesome-module/azurerm"
  version = "1.0.0"
 
  ...
}
 
// Scalr
module "awesome_module_scalr" {
  source = "ministry-of-magic.scalr.io/env-XXX/awesome-module/azurerm"
  version = "~> 1.0"
}

Naming Modules

As we all know naming is a very contentious topic in the realms of IT, that is, unless it is a Terraform module (and you want to host it in a registry). Whilst local modules, or modules that you are going to be sourcing from most remote services can have any name at all if your module is going to be hosted on any registry that implements the Module Registry Protocol will require that your module be named in the terraform-- format. Because of this requirement, your hosting repository must also be named in the same way.

If you do not name your repository in the above format you are going to receive the following error:

{
  "errors": [
    {
      "status": "422",
      "title": "unprocessable entity",
      "detail": "Validation failed: Name is invalid"
    }
  ]
}

If you're going to be moving to a Terraform Cloud/Enterprise, or any other solution that uses the Module Registry Protocol such as Scalr then you should think about setting your modules up with the above naming from the get-go.

File Structure

The file structure of a module is almost identical to the structure of any other Terraform code, there is a small exception in that you should not provide a providers.tf file with your module code as that will come from the caller.

If we were going to create a simple module for an Azure Virtual Machine that could enable Private Link we would structure the repository like so:

.
├── README.md
├── examples/
├── linux.tf
├── network.tf
├── outputs.tf
├── private-link.tf
├── terraform.tf
├── tests/
├── variables.tf
└── windows.tf

We could name the repository in two ways:

With the provider: terraform-azurerm-virtual-machine
With our organization: terraform-mom-virtual-machine

You might consider option two if it was wrapping a few providers.

Let's go through each of these files and talk about what their function is.

README.md — holds documentation about our module and its use, as well as links to relevant information. This will also be shown on the Readme page of the module registry.
examples/ — where example implementations should be to allow quick use, these may also be used as test fixtures.
linux.tf — where the resource declaration occurs for a Linux VM and anything specific to it.
network.tf — all networking setup occurs here irrespective of which OS is used.
outputs.tf — data that we want to return to the caller.
private-link.tf — the optional setup of Private Link for the virtual machine(s).
terraform.tf — where we set our Terraform core and provider requirements.
tests/ — if the module is being tested with something like TerraTest then this is where those tests would reside.
variables.tf — where we declare the interface for our module, this MUST be very explicit and exceptionally well documented as our consumers use this to understand what needs to be passed in.
windows.tf — similarly to our linux.tf this holds configuration specific for a Windows VM if it was selected.

As you can see there are a few files there even for a module that could be considered rather simple and adheres to Phase 3 - The Domain Files (My Terraform Development Workflow). It is exceptionally important when developing modules to be asking yourself these questions:

What does this module provide over using the provider resource?
How does this module make my consumer's life easier?
How does this module make an engineer's life easier to continue development on the module?

If you have a positive answer for all of the above then you meet the criteria for creating a module then you should be going for it. Remember your module should always make someone's life easier otherwise you might want to reconsider.

As an exercise let's answer the above three questions for our Virtual Machine module:

This module provides a simpler interface that can create either a Linux or Windows VM with the ability to enable Private Link in a single module block.
As a consumer of the module you do not need to understand the inner workings of how Private Link works only that you require it to be enabled.
As the module is split out into concise domain files modifying and extending the module is simple, because there is a module we can assume this is a standard within the organization thus if the standard needs to be changed it only has to occur in a single place to propagate out to the consumers.