Advanced Terraform Workflows with terraform_data

This post is part of a series on The Developer's Guide to HCL.

Terraform has become the de facto standard for Infrastructure as Code, allowing teams to define and manage their infrastructure declaratively. While most users are familiar with resources that map directly to cloud components (like an aws_instance), Terraform also offers more abstract tools for complex scenarios. One such tool, introduced in Terraform 1.4, is the terraform_data managed resource.

This built-in resource doesn't create infrastructure itself but serves as a powerful mechanism for managing arbitrary data, orchestrating provisioners, and triggering actions within the Terraform lifecycle. Let's explore how terraform_data works and how you can leverage it for more sophisticated automation.

What is terraform_data?

At its core, terraform_data is a managed resource that lives within your Terraform state. Its primary purpose is to:

1. Store arbitrary data: Values you define are stored in the state and can be referenced elsewhere.
2. Trigger provisioners: It can act as a dedicated resource to run scripts when no other infrastructure resource is a logical fit.
3. Influence resource lifecycles: It can be used with the replace_triggered_by lifecycle argument to force the replacement of other resources based on changes to arbitrary values.

It's always available via the built-in terraform.io/builtin/terraform provider, so there's no need to install or configure an external provider.

Key Arguments and Attributes:

• input (Optional): Accepts any value (string, number, list, map). Changes to this value cause Terraform to plan an update for the terraform_data instance.
• triggers_replace (Optional): Also accepts any value. A change here forces the terraform_data resource to be replaced (destroyed and recreated), which is key for re-running provisioners.
• output (Attribute): Reflects the value of the input argument, making it accessible to other parts of your configuration.
• id (Attribute): A unique ID for the resource instance, typically a UUID.

The Evolution: From null_resource to terraform_data

Before terraform_data, similar functionality was often achieved using the null_resource from the hashicorp/null provider. terraform_data is its built-in successor, offering several advantages:

For new configurations on Terraform 1.4+, terraform_data is the recommended choice.

Core Use Cases with Examples

Let's look at some practical applications.

1. Storing Arbitrary Data with Resource Lifecycle

You can centralize configuration data that needs to be part of the Terraform state:

resource "terraform_data" "configuration_params" {
  input = {
    region        = "us-east-1"
    instance_size = "m5.large"
  }
}
 
resource "aws_instance" "example" {
  ami           = "ami-0c55b31ad2c359908" # Example AMI
  instance_type = terraform_data.configuration_params.output.instance_size
  # ... other configurations
}
 
output "configured_region" {
  value = terraform_data.configuration_params.output.region
}

A change to terraform_data.configuration_params.input will update this resource, and potentially the aws_instance if it consumes the output.

2. Triggering Provisioners

When you need to run scripts (e.g., local-exec or remote-exec) that aren't tied to a specific piece of infrastructure, terraform_data is an ideal host:

resource "aws_instance" "web" {
  # ... configuration for web server
}
 
resource "aws_db_instance" "database" {
  # ... configuration for database
}
 
resource "terraform_data" "bootstrap_application" {
  triggers_replace = [
    aws_instance.web.id,
    aws_db_instance.database.id,
  ]
 
  provisioner "local-exec" {
    command = "./scripts/deploy_app.sh ${aws_instance.web.public_ip} ${aws_db_instance.address}"
  }
}

Here, if the web or database instance ID changes (signifying replacement), the terraform_data.bootstrap_application resource is also replaced, re-running the deployment script. While powerful, remember that provisioners should generally be a last resort; prefer managing resources declaratively where possible.

Advanced Patterns

Integrating with replace_triggered_by

A highly effective pattern is using terraform_data to trigger the replacement of another resource based on arbitrary conditions:

variable "app_version" {
  type    = string
  default = "1.0.0"
}
 
resource "terraform_data" "version_tracker" {
  input = var.app_version
}
 
resource "aws_ecs_service" "my_app_service" {
  name            = "my-app"
  cluster         = aws_ecs_cluster.my_cluster.id
  task_definition = aws_ecs_task_definition.my_app_task.arn
  desired_count   = 3
  # ... other configurations
 
  lifecycle {
    replace_triggered_by = [
      terraform_data.version_tracker
    ]
  }
}

If var.app_version changes, terraform_data.version_tracker is updated. The lifecycle block in aws_ecs_service.my_app_service detects this change and plans a replacement for the service, effectively rolling out the new version.

Managing such complex dependencies and lifecycles can become challenging in larger organizations. Platforms like Scalr can provide crucial visibility and governance over these sophisticated Terraform configurations, helping ensure they align with operational best practices and organizational policies through features like customizable OPA policies.

Using terraform_data with for_each and count

You can create multiple terraform_data instances using for_each or count. However, be cautious when using a collection of terraform_data resources in replace_triggered_by for another resource collection. A change in one terraform_data instance might inadvertently trigger the replacement of all instances in the dependent collection. This requires careful testing.

When dealing with iterated resources and such nuanced trigger mechanisms, maintaining control and understanding the potential blast radius of changes is critical. Tools that offer robust environment management and detailed run previews, like those found in Scalr, can offer better insights before applying potentially widespread changes.

Best Practices and Considerations

• When to Use: Ideal for triggering provisioners without a natural host resource, storing lifecycle-bound data, or as an intermediary for replace_triggered_by.
• Sensitive Data: Data in input is stored in plain text in the Terraform state. Secure your state backend rigorously. Provisioners should fetch secrets from secure stores (e.g., HashiCorp Vault, AWS Secrets Manager) at runtime rather than receiving them as direct inputs.
• Idempotency: Ensure provisioner scripts are idempotent (running them multiple times yields the same result without errors).
• Unknown Values: If input depends on a yet-to-be-created resource, its output will be unknown during the plan phase. This can cause issues if used in count or for_each of other resources.
• Avoid Overuse: Don't let terraform_data become a crutch for overly complex imperative logic. Strive for declarative configurations.

As configurations grow with advanced resources like terraform_data, enforcing security policies (e.g., around sensitive data handling or provisioner usage) and maintaining operational consistency becomes paramount. Integrating policy-as-code frameworks, such as Open Policy Agent (OPA) managed through platforms like Scalr, allows organizations to codify and automatically enforce these standards across all Terraform operations.

terraform_data vs. locals

It's important to distinguish terraform_data from local values (locals):

• locals: Compile-time conveniences for naming expressions. They don't have a lifecycle and aren't stored independently in the state. They cannot directly trigger provisioners or be used in replace_triggered_by.
• terraform_data: Actual resources with a lifecycle, persisted in the state. They can trigger provisioners and be used in replace_triggered_by.

Use locals for simplifying expressions and terraform_data when you need resource-like behavior for data or actions.

Conclusion

The terraform_data resource is a versatile tool in the modern Terraform practitioner's arsenal. It provides elegant solutions for managing arbitrary data within a resource lifecycle, orchestrating provisioners, and implementing sophisticated triggering mechanisms like replace_triggered_by.

While it unlocks advanced automation patterns, its power comes with the need for careful design. Understanding its nuances, potential pitfalls (especially with collections and sensitive data), and adhering to best practices will ensure you leverage terraform_data effectively, leading to more robust and maintainable Infrastructure as Code. As your Terraform usage matures, consider how comprehensive IaC management platforms can help you scale these advanced practices with confidence and control.