Using the AWS S3 Backend Block in Terraform

If you manage AWS infrastructure with Terraform, sooner or later you need a remote backend. The s3 backend block is the usual way to do it. It keeps your Terraform state files in an Amazon S3 bucket so a team can share them, lock them during a run, and keep the state file from getting corrupted.

The Terraform state file is a JSON record of what you've deployed. It maps your configuration to the actual resources in your AWS account. Keeping that file in S3 instead of on one person's laptop buys you a few things:

• Collaboration: It allows multiple team members to work on the same infrastructure without overwriting each other's changes.
• State Locking: It prevents concurrent terraform apply operations, which can lead to state file corruption and resource conflicts.
• Security & Durability: State files can contain sensitive data. Storing them in a secure, encrypted, and highly available S3 bucket is a core best practice.

---

Basic Usage and Configuration

To use the s3 backend, you need a pre-existing Amazon S3 bucket. For state locking, it's also a best practice to use a DynamoDB table.

Here's a basic s3 backend block configuration:

terraform {
  backend "s3" {
    bucket         = "my-terraform-state-bucket"
    key            = "my-app.tfstate"
    region         = "us-east-1"
    dynamodb_table = "terraform-locks"
    encrypt        = true
  }
}

• bucket: The globally unique name of your S3 bucket.
• key: The name of the object (the state file itself) within the bucket. You should use a unique key for each separate Terraform configuration.
• region: The AWS region where your S3 bucket is located.
• dynamodb_table: The name of the DynamoDB table used for state locking. This prevents concurrent writes.
• encrypt: A boolean value that enables server-side encryption for the state file at rest.

After adding this block to your main Terraform configuration file, you must run terraform init. This command initializes the backend and prompts you to migrate any existing local state to the remote S3 bucket.

---

Use Cases and Best Practices

The s3 backend is essential for any production Terraform project.

• Team Projects: When multiple developers are working on a single infrastructure, the s3 backend ensures everyone is using the same, up-to-date state file.
• CI/CD Pipelines: In a continuous integration and continuous deployment (CI/CD) workflow, the pipeline needs a consistent place to read and write the state file. The s3 backend provides a reliable and secure endpoint for tools like AWS CodePipeline or GitHub Actions to execute Terraform.
• Production Environments: For production infrastructure, the s3 backend is non-negotiable. Its built-in state locking and data durability features are critical for preventing downtime and ensuring the integrity of your production environment.

Best Practices:

• Separate State: Use a separate S3 bucket and DynamoDB table for each environment (e.g., dev, stage, prod). This isolates state files and prevents accidental cross-environment changes.
• Permission Scoping: Use IAM policies to grant the minimum required permissions to your users or roles. They should only have permissions to read from and write to their specific state bucket and locking table.
• Versioning: Enable versioning on your S3 bucket. This gives you a history of your state files and allows you to revert to a previous version if something goes wrong.

---

Authentication and Examples

You should never hardcode credentials like access keys directly in your configuration. Instead, use one of the supported authentication methods:

Environment Variables: The most common approach. Terraform can automatically use credentials set in environment variables like AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.

export AWS_ACCESS_KEY_ID="<your-access-key-id>"
export AWS_SECRET_ACCESS_KEY="<your-secret-access-key>"
export AWS_DEFAULT_REGION="us-east-1"

terraform init

AWS CLI: If you're already logged in via aws configure, Terraform picks up those credentials on its own. This is the easiest option for local development. The CLI caches a token and credentials when you set it up, and Terraform reads them from there.

Configure the AWS CLI on your machine.

aws configure
AWS Access Key ID [None]: <your-access-key-id>
AWS Secret Access Key [None]: <your-secret-access-key>
Default region name [None]: us-east-1
Default output format [None]: json

After this is complete, you can run Terraform commands without any extra authentication configuration.

terraform init

Terraform will automatically find the cached credentials and use them.

IAM Roles: The most secure method for resources running in AWS. An EC2 instance, Lambda function, or CodeBuild job can assume an IAM role with the permissions it needs to reach the S3 bucket.

You attach an IAM role to the resource, and Terraform picks up that role's permissions automatically.

1. Create an IAM Role with a trust policy that allows a specific service (e.g., ec2.amazonaws.com or codebuild.amazonaws.com) to assume it.
2. Attach a policy to the role that grants permissions to access the S3 bucket and DynamoDB table. For example, a policy might look like this:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:*",
      "Resource": "arn:aws:s3:::my-terraform-state-bucket/*"
    },
    {
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::my-terraform-state-bucket"
    },
    {
      "Effect": "Allow",
      "Action": "dynamodb:*",
      "Resource": "arn:aws:dynamodb:us-east-1:123456789012:table/terraform-locks"
    }
  ]
}

3. Attach the IAM Role to your EC2 instance profile, Lambda function, or CodeBuild project. Terraform, when run on that resource, will automatically authenticate using the temporary credentials provided by the role. No secrets are stored on the resource itself.

Service Principal: For CI/CD pipelines running on platforms outside of AWS, a dedicated IAM user with limited permissions is the standard approach. Store the access key and secret in your CI/CD platform's secrets manager.

1. Create a dedicated IAM User for your CI/CD pipeline.
2. Generate an Access Key and Secret Key for that user. Note: This is the only time you will see the secret key, so you must save it immediately.
3. Attach an IAM Policy to the user with the required permissions to access your S3 bucket and DynamoDB table, just as in the IAM Roles example above.
4. Store the Access Key and Secret Key as secure variables or secrets in your CI/CD platform's secrets manager. You should never commit these credentials to your code repository.
5. In your pipeline configuration, set these secrets as environment variables before running Terraform. This ensures your credentials are never exposed in logs or source code.

---

Advanced Concepts

Partial Backend Configuration

Terraform's design prevents you from using variables directly inside the backend block. However, you can leave out sensitive or environment-specific information and supply it at runtime using a backend configuration file or command-line flags with terraform init.

Example using a file:

Run terraform init:

terraform init -backend-config="backend.conf"

backend.conf**:**

region = "us-east-1"
dynamodb_table = "terraform-locks"

main.tf (partial config):

terraform {
  backend "s3" {
    bucket = "my-terraform-state-bucket"
    key    = "my-app.tfstate"
  }
}

This method prevents sensitive information from being committed to source control.

Managing Multiple Environments with Workspaces

For a single configuration that deploys to multiple environments, Terraform workspaces can be used to manage different state files within the same bucket.

Example:

# Create and switch to a development workspace
terraform workspace new dev

# Create and switch to a production workspace
terraform workspace new prod

When you switch between workspaces, Terraform automatically changes the key to include the workspace name (e.g., env:/dev/my-app.tfstate), ensuring each environment has its own isolated state file.

---

A Note on OpenTofu and Dynamic Backends

The Terraform CLI still forbids dynamic backend blocks. OpenTofu, a fork of Terraform, does not.

OpenTofu, starting with version 1.8, added the ability to use variables and local values inside the backend block. People had been asking for this in the Terraform community for years, so it's a big deal.

That lets you write a more flexible, DRY backend configuration, which helps most when you're managing several environments.

Here's how a dynamic s3 backend block could look in OpenTofu:

variable "env" {
  type    = string
  default = "dev"
}

terraform {
  backend "s3" {
    bucket         = "my-terraform-state-${var.env}-bucket"
    key            = "my-app.tfstate"
    region         = "us-east-1"
    dynamodb_table = "terraform-locks-${var.env}"
    encrypt        = true
  }
}

Now you can switch environments by changing the env variable, which you can pass at the command line:

tofu init -var="env=prod"

You no longer need separate backend configuration files or scripts to juggle environments, so there's less to get wrong by hand. If you run many similar environments, or you just prefer a variable-driven setup, OpenTofu's dynamic backend blocks save you real work.

Want to learn more about other backends? Check out the links below:

• Azure: /learning-center/using-the-azurerm-backend-block-in-terraform/
• GCS: /learning-center/using-the-gcs-backend-block-in-terraform/

This blog has been verified for Terraform and OpenTofu