Using the GCS Backend Block in Terraform

If you manage Google Cloud infrastructure with Terraform, sooner or later you need a remote backend. The GCS backend block is the usual way to do it: it stores your Terraform state files in a Google Cloud Storage bucket so a team can share them safely.

The Terraform state file is a JSON record of the resources you've deployed. It maps your configuration to the actual resources in your Google Cloud project. Keeping that file in a GCS bucket instead of on a laptop gets you a few things:

• Collaboration: Several team members can work on the same infrastructure without overwriting each other's changes.
• State Locking: Concurrent terraform apply operations are blocked, since they can corrupt the state file and cause resource conflicts. Google Cloud Storage handles this locking automatically.
• Security & Durability: State files can contain sensitive data, so they belong in an encrypted, highly available GCS bucket.

---

How Do You Configure a Basic GCS Backend Block?

To use the GCS backend, you need a Google Cloud Storage bucket that already exists. It's a good idea to create a dedicated bucket for your Terraform state files.

Here's a basic GCS backend block configuration:

terraform {
  backend "gcs" {
    bucket = "my-terraform-state-bucket"
    prefix = "my-app"
  }
}

• bucket: The globally unique name of your GCS bucket.
• prefix: The path within the bucket where the state file will be stored. The state file will be named <prefix>/terraform.tfstate.

After adding this block to your main Terraform configuration file, run terraform init. It initializes the backend and asks whether to migrate any existing local state to the GCS bucket.

---

When Should You Use the GCS Backend, and How?

Once you're past a single-person experiment, the GCS backend earns its keep in a few common situations:

• Team Projects: When several developers work on the same infrastructure, the GCS backend keeps everyone on the same, up-to-date state file.
• CI/CD Pipelines: A CI/CD pipeline needs a consistent place to read and write the state file. The GCS backend gives tools like Cloud Build or GitHub Actions that place when they run Terraform.
• Production Environments: For production infrastructure, don't skip it. Its built-in state locking and data durability keep you from losing data or corrupting the state of your production environment.

Best Practices:

• Separate State: Use a separate GCS bucket or a unique prefix for each environment. This isolates state files and prevents accidental cross-environment changes.
• Permission Scoping: Use IAM policies to grant the minimum required permissions to your users or service accounts. They should only have permissions to read from and write to their specific state bucket.
• Enable versioning on your GCS bucket: This gives you a history of your state files and allows you to revert to a previous version if something goes wrong.

---

How Do You Authenticate Terraform to Access the GCS Bucket?

Never hardcode credentials like service account keys in your configuration. Use one of the supported authentication methods instead:

• gcloud CLI: The most common approach for local development. When you log in with the gcloud CLI, Terraform can use those credentials automatically.
• Environment Variables: Terraform can use credentials set in environment variables like GOOGLE_APPLICATION_CREDENTIALS which points to a service account key file.
• Service Accounts: The most secure method for resources running in GCP. A Compute Engine instance or Cloud Build job can use a service account with the necessary permissions to access the GCS bucket.
• Workload Identity: The recommended way to authenticate for applications running on Google Cloud, especially with GKE, as it links a Kubernetes service account to a Google Cloud service account.

Examples

gcloud CLI

This is the easiest way to get started with local development. When you authenticate with the gcloud CLI, it sets up Application Default Credentials (ADC) which Terraform uses automatically.

Run Terraform commands without any extra authentication configuration:

terraform init

Set your project (if you have multiple):

gcloud config set project <your-project-id>

Log in to GCP:

gcloud auth application-default login

Environment Variables

This approach keeps sensitive information out of your code. You can download a service account key file and point to it using an environment variable.

export GOOGLE_APPLICATION_CREDENTIALS="/path/to/your/keyfile.json"

terraform init

Service Accounts

This is the most secure method for authenticating resources running inside GCP, such as a Compute Engine instance or Cloud Build job. Attach a service account to the resource, and Terraform assumes that service account's permissions automatically.

1. Create a Service Account with a role that grants permissions to access the GCS bucket. The Storage Object Admin role is a common choice.
2. Attach the Service Account to your Compute Engine instance or Cloud Build project. When Terraform runs on that resource, it authenticates with the service account's credentials automatically, and no secrets are stored on the resource itself.

---

What Advanced GCS Backend Configurations Should You Know?

Partial Backend Configuration

Terraform won't let you use variables directly inside the backend block. But you can leave out sensitive or environment-specific values and supply them at runtime with a backend configuration file or command-line flags on terraform init.

Example using a file:

Run terraform init:

terraform init -backend-config="backend.conf"

backend.conf (sensitive info):

bucket = "my-terraform-state-bucket"

main.tf (partial config):

terraform {
  backend "gcs" {
    prefix = "my-app"
  }
}

This keeps sensitive values out of source control.

Managing Multiple Environments with Workspaces

If one configuration deploys to several environments, you can use Terraform workspaces to keep a separate state file for each one inside the same bucket.

Example:

# Create and switch to a development workspace
terraform workspace new dev

# Create and switch to a production workspace
terraform workspace new prod

When you switch between workspaces, Terraform automatically changes the prefix to include the workspace name (e.g., my-app/env:/dev/terraform.tfstate), ensuring each environment has its own isolated state file.

---

A Note on OpenTofu and Dynamic Backends

The classic Terraform CLI still refuses to let you put variables in a backend block. OpenTofu, the fork of Terraform, took a different path here.

Starting with version 1.8, OpenTofu lets you use variables and local values inside the backend block. That answers one of the oldest feature requests in the Terraform community, and it lets you write a DRY (Don't Repeat Yourself) backend configuration, which helps most when you run the same setup across several environments.

Here's how a dynamic gcs backend block could look in OpenTofu:

variable "env" {
  type    = string
  default = "dev"
}

terraform {
  backend "gcs" {
    bucket = "my-terraform-state-${var.env}-bucket"
    prefix = "my-app-${var.env}"
  }
}

Now you switch environments by changing the env variable. You can pass it at the command line:

tofu init -var="env=prod"

That cuts out the separate backend config files or wrapper scripts you'd otherwise write to juggle environments, so there's less to keep in sync by hand. If you manage a lot of near-identical environments, OpenTofu's dynamic backend blocks are worth a look.

Standing up the bucket, locking, and IAM yourself is fine for a single team. Once several teams share state, a managed platform that provides the remote backend and run execution removes that upkeep. Scalr runs Terraform and OpenTofu with state and locking built in on usage-based pricing that's free up to 50 runs a month.

This blog has been verified for Terraform and OpenTofu.