Terraform Configuration Ingestion

Key takeaways

Duplicated Terraform code across environments can be eliminated by ingesting configuration three ways: variable files, higher-order modules, or YAML/JSON configuration files.
Variable files (.tfvars) require converting static assignments to input variables, then supplying a separate tfvars file per environment passed in via the CI/CD pipeline.
Turning code into a higher-order module lets you version and control what each environment consumes, and the module itself can be driven by variable or configuration files.
YAML or JSON config files, read with yamldecode or jsondecode, co-locate multiple projects and environments in one place but carry blast-radius, validation, and structure tradeoffs.

One problem I see pretty regularly with Terraform codebases is a lot of repeated code. That repeated code is usually there to deploy environments or multiple instances of the same codebase. Picture some networking and a virtual machine that hosts a web application. The code gets copied and pasted into different folders or repositories, with some (usually minor) configuration changed. If you had three environments, you now have three copies of the codebase to maintain and keep in sync, which puts extra and unnecessary strain on your engineers. So how would I fix this?

The answer might seem easy, but in my experience it ends up being a little more complicated. It starts with removing the repeated, often near-identical code. Realistically your non-production environments should be identical to production, just at a reduced scale. You can solve all of this by ingesting configuration into your Terraform codebase or module. You wouldn't keep three copies of your application's code, now would you?

In my mind there are three ways that this can be achieved:

Terraform Variable Files
Usage of higher-order modules
yaml/json configuration Files

I'll go through each one in turn. First, here's the code you'd write in the copy-and-paste scenario, which we'll then rework for each ingestion approach.

First we set up our providers.

provider.tf

provider "azurerm" {
  features {}
}

Next we'll set up the networking infrastructure. Notice that every value here is statically assigned.

network.tf

resource "azurerm_resource_group" "this" {
  name     = format("rg-%s", local.name_suffix)
  location = "australiaeast"
}
 
resource "azurerm_virtual_network" "this" {
  name                = format("vn-%s", local.name_suffix)
  location            = azurerm_resource_group.this.location
  resource_group_name = azurerm_resource_group.this.name
 
  address_space = ["10.0.0.0/23"]
}
 
resource "azurerm_subnet" "this" {
  name                 = format("sn-%s", local.name_suffix)
  resource_group_name  = azurerm_resource_group.this.name
  virtual_network_name = azurerm_virtual_network.this.name
 
  address_prefixes = ["10.0.1.0/24"]
}
 
resource "azurerm_public_ip" "this" {
  name                = format("pip-%s", local.name_suffix)
  location            = azurerm_resource_group.this.location
  resource_group_name = azurerm_resource_group.this.name
 
  allocation_method = "Dynamic"
}
 
resource "azurerm_network_security_group" "this" {
  name                = format("nsg-%s", local.name_suffix)
  location            = azurerm_resource_group.this.location
  resource_group_name = azurerm_resource_group.this.name
}
 
resource "azurerm_subnet_network_security_group_association" "this" {
  subnet_id                 = azurerm_subnet.this.id
  network_security_group_id = azurerm_network_security_group.this.id
}
 
resource "azurerm_network_interface" "this" {
  name                = format("nic-%s", local.name_suffix)
  location            = azurerm_resource_group.this.location
  resource_group_name = azurerm_resource_group.this.name
 
  ip_configuration {
    name                          = "config"
    subnet_id                     = azurerm_subnet.this.id
    private_ip_address_allocation = "Dynamic"
    public_ip_address_id          = azurerm_public_ip.this.id
  }
}

Create our naming suffix in the form of a local variable, SSH key, and the virtual machine.

main.tf

locals {
  name_suffix = "aue-dev-blt"
}
 
resource "tls_private_key" "this" {
  algorithm = "RSA"
  rsa_bits  = "4096"
}
 
resource "azurerm_linux_virtual_machine" "this" {
  name                = replace(format("vm-%s", local.name_suffix), "-", "")
  location            = azurerm_resource_group.this.location
  resource_group_name = azurerm_resource_group.this.name
 
  size           = "Standard_F2"
  admin_username = "adminuser"
  network_interface_ids = [
    azurerm_network_interface.this.id
  ]
 
  admin_ssh_key {
    username   = "adminuser"
    public_key = tls_private_key.this.public_key_openssh
  }
 
  os_disk {
    caching              = "ReadWrite"
    storage_account_type = "Standard_LRS"
  }
 
  source_image_reference {
    publisher = "Canonical"
    offer     = "UbuntuServer"
    sku       = "16.04-LTS"
    version   = "latest"
  }
}

Finally we'll output our private key so that we can access the server!

outputs.tf

output "ssh_private_key" {
  value     = tls_private_key.this.private_key_openssh
  sensitive = true
}

Warning! Do not do this in any real-world scenario. Put it in a secrets vault of some description!

How Do You Use Terraform tfvars Files to Remove Duplication?

To use tfvars files we first need input variables, so the first change is to remove all that static assignment and use variables instead!

variables.tf

variable "location" {
  type        = string
  description = <<EOT
  (Required)  The Azure region where the resources will be deployed.
  EOT
 
  validation {
    condition = contains(
      ["australiaeast", "australiasoutheast"],
      var.location
    )
    error_message = "Err: invalid Azure location provided."
  }
}
 
variable "environment" {
  type        = string
  description = <<EOT
  (Required)  The environment short name for the resources.
  EOT
 
  validation {
    condition = contains(
      ["dev", "uat", "prd"],
      var.environment
    )
    error_message = "Err: invalid environment provided."
  }
}
 
variable "project_identifier" {
  type        = string
  description = <<EOT
  (Required)  The identifier for the project, 4 character maximum
  EOT
 
  validation {
    condition     = length(var.project_identifier) <= 4
    error_message = "Err: project identifier cannot be longer than 4 characters."
  }
}
 
variable "network" {
  type = object({
    address_space = string
    subnets = list(
      object({
        role          = string
        address_space = string
      })
    )
  })
  description = <<EOT
  (Required)  Network details
  EOT
}

Above we've set up variables for location, environment, project_identifier and network. In a real-world scenario we'd likely pass in a lot of the Virtual Machine configuration as well, but to keep the code simple we'll stick to the bare minimum. With those inputs in place, the codebase needs updating to use them.

main.tf

locals {
  location_map = {
    "australiaeast"      = "aue"
    "australiasoutheast" = "aus"
  }
 
  name_suffix = lower(format(
    "%s-%s-%s",
    local.location_map[var.location],
    var.environment,
    var.project_identifier
  ))
}
 
resource "azurerm_resource_group" "this" {
  name     = format("rg-%s", local.name_suffix)
  location = var.location
}
 
resource "tls_private_key" "this" {
  algorithm = "RSA"
  rsa_bits  = "4096"
}
 
resource "azurerm_linux_virtual_machine" "this" {
  name                = replace(format("vm-%s", local.name_suffix), "-", "")
  location            = azurerm_resource_group.this.location
  resource_group_name = azurerm_resource_group.this.name
 
  size           = "Standard_F2"
  admin_username = "adminuser"
  network_interface_ids = [
    azurerm_network_interface.this.id
  ]
 
  admin_ssh_key {
    username   = "adminuser"
    public_key = tls_private_key.this.public_key_openssh
  }
 
  os_disk {
    caching              = "ReadWrite"
    storage_account_type = "Standard_LRS"
  }
 
  source_image_reference {
    publisher = "Canonical"
    offer     = "UbuntuServer"
    sku       = "16.04-LTS"
    version   = "latest"
  }
}

The locals block is a little more interesting now. The location input variable selects the short name for the location, which we use when naming resources. Because the suffix is built from input variables, changing the environment only means changing one input.

network.tf

resource "azurerm_virtual_network" "this" {
  name                = format("vn-%s", local.name_suffix)
  location            = azurerm_resource_group.this.location
  resource_group_name = azurerm_resource_group.this.name
 
  address_space = [var.network.address_space]
}
 
resource "azurerm_subnet" "this" {
  for_each = {
    for v in var.network.subnets :
    v.role => v.address_space
  }
 
  name                 = format("sn-%s-%s", local.name_suffix, each.key)
  resource_group_name  = azurerm_resource_group.this.name
  virtual_network_name = azurerm_virtual_network.this.name
 
  address_prefixes = [each.value]
}
 
resource "azurerm_public_ip" "this" {
  name                = format("pip-%s", local.name_suffix)
  location            = azurerm_resource_group.this.location
  resource_group_name = azurerm_resource_group.this.name
 
  allocation_method = "Dynamic"
}
 
resource "azurerm_network_security_group" "this" {
  name                = format("nsg-%s", local.name_suffix)
  location            = azurerm_resource_group.this.location
  resource_group_name = azurerm_resource_group.this.name
}
 
resource "azurerm_subnet_network_security_group_association" "this" {
  for_each = azurerm_subnet.this
 
  subnet_id                 = each.value.id
  network_security_group_id = azurerm_network_security_group.this.id
}
 
resource "azurerm_network_interface" "this" {
  name                = format("nic-%s", local.name_suffix)
  location            = azurerm_resource_group.this.location
  resource_group_name = azurerm_resource_group.this.name
 
  ip_configuration {
    name                          = "config"
    subnet_id                     = azurerm_subnet.this["iaas"].id
    private_ip_address_allocation = "Dynamic"
    public_ip_address_id          = azurerm_public_ip.this.id
  }
}

The network input variable now feeds the network configuration, and it can create multiple subnets if we want them.

So how do we consume this codebase? With tfvar files: you'd have one tfvar file for every environment or project you want to build. Here's an example:

dev.tfvars

location           = "australiaeast"
environment        = "dev"
project_identifier = "blt"
network = {
  address_space = "10.0.0.0/23"
  subnets = [
    {
      role          = "iaas"
      address_space = "10.0.1.0/24"
    }
  ]
}

prd.tfvars

location           = "australiaeast"
environment        = "prd"
project_identifier = "blt"
network = {
  address_space = "10.10.0.0/23"
  subnets = [
    {
      role          = "iaas"
      address_space = "10.10.1.0/24"
    }
  ]
}

We have two environments declared above. They completely change the configuration of the Terraform code without touching the code itself... no more copy and paste! Your CI/CD pipeline would then select the correct tfvars file and pass it into the Terraform run. Now a change to the codebase flows through your environments more easily, and when you're testing or developing in the lower environments you can be fairly confident that what works in development will work in production too.

Why Turn Your Terraform Code Into a Higher-Order Module?

Now that we've defined input variables for our code, we can go one step further and turn that code into a module. This lets us strictly version and control which version of our code each environment consumes. We could drive the module with either variable files or configuration files. For this example, we'll statically assign the inputs.

First, create the module itself locally.

$ mkdir -p $(pwd)/modules/app-server

Then create the files that will hold the code:

touch -p $(pwd)/modules/app-server/{main,network,inputs,outputs}.tf

You might notice there's no providers.tf file. The module doesn't need one: when we call the module, we create an instance of the correct provider at that level and Terraform sorts it out for us.

So this post doesn't run to 10,000 lines, I won't include the module code itself. It's identical to the code in the Variable Files section, just placed in the empty files we created. Here's how we call the module instead.

main.tf

provider "azurerm" {
  features {}
}
 
module "dev_app_server" {
  source = "./modules/app-server"
 
  location           = "australiaeast"
  environment        = "dev"
  project_identifier = "blt"
  network = {
    address_space = "10.0.0.0/23"
    subnets = [
      {
        role          = "iaas"
        address_space = "10.0.1.0/24"
      }
    ]
  }
 
}
 
output "ssh_private_key" {
  value     = module.dev_app_server.ssh_private_key
  sensitive = true
}

The interface for the module is simple, and it's satisfied by passing through everything we declared as input variables. Modules give you a lot of control over what the consumers of your code are doing. Here the module is defined locally, but nothing stops us from storing it in git or a module registry like Terraform Cloud.

Creating a module also gives our consumers more self-service. Everything they need to create an app-server is defined here, and they know it satisfies organisational requirements like security controls.

How Does yamldecode Feed YAML or JSON Configuration Into Terraform?

The final way to pass configuration into our Terraform code is with configuration files. These are json or yaml files that we ingest using the yamldecode or jsondecode functions, which gives us dot access to the attributes.

Here's the example yaml file we'll use:

config.yaml

project:
  blt:
    environments:
      dev:
        location: australiaeast
        network:
          address_space: 10.0.0.0/23
          subnets:
          - role: iaas
            address_space: 10.0.1.0/24
      prd:
        location: australiaeast
        network:
          address_space: 10.1.0.0/23
          subnets:
          - role: iaas
            address_space: 10.1.1.0/24

A lot of the keys here will look familiar, since they represent the input variables we were previously passing into our Terraform code. The advantage of the configuration file is that it can house multiple projects in a single file, with all their environments alongside. We could even turn the configuration file into a module, or a provider!

Now for the code, and how it consumes this configuration file.

variables.tf is smaller now. It only holds what I call orientation variables, because they let our Terraform code orient itself within the configuration file.

variables.tf

variable "environment" {
  type        = string
  description = <<EOT
  (Required)  The environment short name for the resources.
  EOT
 
  validation {
    condition = contains(
      ["dev", "uat", "prd"],
      var.environment
    )
    error_message = "Err: invalid environment provided."
  }
}
 
variable "project_identifier" {
  type        = string
  description = <<EOT
  (Required)  The identifier for the project, 4 character maximum
  EOT
 
  validation {
    condition     = length(var.project_identifier) <= 4
    error_message = "Err: project identifier cannot be longer than 4 characters."
  }
}

Next is main.tf, which does the work of consuming the configuration file. Two local variables, raw_config and config, set up our environment.

raw_config: we ingest the config file itself into Terraform and then we use the yamldecode function to convert the yaml into configuration that Terraform can easily work with.
config: using our orientation variables and passing those into some lookup functions we set our local.config to be scoped to the project blt for the dev environment.

main.tf

locals {
  raw_config = yamldecode(file("./config.yaml"))
  config = lookup(
    lookup(
      local.raw_config.project,
      var.project_identifier,
      null
    ).environments,
    var.environment,
    null
  )
 
  location_map = {
    "australiaeast"      = "aue"
    "australiasoutheast" = "aus"
  }
 
  name_suffix = lower(format(
    "%s-%s-%s",
    local.location_map[local.config.location],
    var.environment,
    var.project_identifier
  ))
}
 
...

The last file is network.tf. I've truncated the unchanged, boring portions here too.

network.tf

resource "azurerm_virtual_network" "this" {
  name                = format("vn-%s", local.name_suffix)
  location            = azurerm_resource_group.this.location
  resource_group_name = azurerm_resource_group.this.name
 
  address_space = [local.config.network.address_space]
}
 
resource "azurerm_subnet" "this" {
  for_each = {
    for v in local.config.network.subnets :
    v.role => v.address_space
  }
 
  name                 = format("sn-%s-%s", local.name_suffix, each.key)
  resource_group_name  = azurerm_resource_group.this.name
  virtual_network_name = azurerm_virtual_network.this.name
 
  address_prefixes = [each.value]
}
 
...

The part to notice is that the subnets object now comes from local.config. Here's the for loop with the config file next to the earlier version with variables:

// Config
resource "azurerm_subnet" "this" {
  for_each = {
    for v in local.config.network.subnets :
    v.role => v.address_space
  }
  ...
}
 
// Variables
resource "azurerm_subnet" "this" {
  for_each = {
    for v in var.network.subnets :
    v.role => v.address_space
  }
  ...
}

The two look almost identical, so why bother with configuration files, I hear you ask? I've found them useful, especially in larger and more complex projects, because you can co-locate a lot of configuration in one place and then consume it from many. Say you pull all the networking out to be handled by a central networking component, but you still want information about that infrastructure. You can read it from the configuration file. Maybe you consume the configuration directly, or maybe it gives you enough to construct data sources. Either way it removes the dependency on the other workspace or codebase. Or say you have a cost center tag that needs to be on every resource in the blt project. Those resources might be scattered across any number of Terraform codebases, and with the configuration file you can update that value everywhere from a single place.

This approach does come with trade-offs:

Blast Radius: one shared source cuts both ways. If someone mistakenly changes a line of configuration, it affects all code consuming that configuration.
Validation: because we're now consuming yaml or json, the codebase needs some validation so an invalid value (e.g. null) doesn't break it. I think that's a benefit in the long run; it just costs some development time upfront.
Structure: you also have to settle on a structure for your configuration and enforce it for all consumers, or you could break things. I'd solve this with easy-to-consume documentation and/or json/yaml schema files.

Which Terraform Configuration Ingestion Method Should You Use?

We've gone through three ways of ingesting configuration into Terraform code. The most basic is the tfvar file, which pre-fills the required input variables and passes different instances of those inputs into the same Terraform code. A bit more advanced is the higher-order module, where you wrap your Terraform code in a module and provide an interface for your consumers to fill. Last is the configuration file, such as a yaml file. It gives you the most flexibility, and you can pair it with the higher-order module too.

There's no rule that says "you must use Method A every time because reason X". Each has different benefits, so pick the one that suits your situation.

Hopefully that gives you a useful starting point for getting configuration into your Terraform code and cutting down on duplication. As always, if you have feedback or want to know more, feel free to reach out!

You can follow Brendan @BrendanLiamT on Twitter.

About the author

Brendan Thompson

solutions engineer at Scalr

Brendan Thompson is a solutions engineer at Scalr, specializing in Terraform and cloud infrastructure.

All articles

Part of this guide

25 sheets

CI/CD and GitOps for Terraform & OpenTofu

24 articlesSep 2026