Using the GCS Backend Block in Terraform

If you manage Google Cloud infrastructure with Terraform, sooner or later you need a remote backend. The GCS backend block is the usual way to do it: it stores your Terraform state files in a Google Cloud Storage bucket so a team can share them safely.

The Terraform state file is a JSON record of the resources you've deployed. It maps your configuration to the actual resources in your Google Cloud project. Keeping that file in a GCS bucket instead of on a laptop gets you a few things:

• Collaboration: It lets multiple team members work on the same infrastructure without overwriting each other's changes.
• State Locking: It prevents concurrent terraform apply operations, which can lead to state file corruption and resource conflicts. Google Cloud Storage handles this locking automatically.
• Security & Durability: State files can contain sensitive data. Storing them in a secure, encrypted, and highly available GCS bucket is a core best practice.

---

Basic Usage and Configuration

To use the GCS backend, you need a Google Cloud Storage bucket that already exists. It's a good idea to create a dedicated bucket for your Terraform state files.

Here's a basic GCS backend block configuration:

terraform {
  backend "gcs" {
    bucket = "my-terraform-state-bucket"
    prefix = "my-app"
  }
}

• bucket: The globally unique name of your GCS bucket.
• prefix: The path within the bucket where the state file will be stored. The state file will be named <prefix>/terraform.tfstate.

After adding this block to your main Terraform configuration file, you must run terraform init. This command initializes the backend and prompts you to migrate any existing local state to the remote GCS bucket.

---

Use Cases and Best Practices

Once you're past a single-person experiment, the GCS backend earns its keep in a few common situations:

• Team Projects: When multiple developers are working on a single infrastructure, the GCS backend ensures everyone is using the same, up-to-date state file.
• CI/CD Pipelines: In a continuous integration and continuous deployment (CI/CD) workflow, the pipeline needs a consistent place to read and write the state file. The GCS backend gives tools like Cloud Build or GitHub Actions a reliable, secure place to run Terraform.
• Production Environments: For production infrastructure, the GCS backend is non-negotiable. Its built-in state locking and data durability keep you from losing data or corrupting the state of your production environment.

Best Practices:

• Separate State: Use a separate GCS bucket or a unique prefix for each environment. This isolates state files and prevents accidental cross-environment changes.
• Permission Scoping: Use IAM policies to grant the minimum required permissions to your users or service accounts. They should only have permissions to read from and write to their specific state bucket.
• Enable versioning on your GCS bucket: This gives you a history of your state files and allows you to revert to a previous version if something goes wrong.

---

Authentication and Examples

You should never hardcode credentials like service account keys directly in your configuration. Instead, use one of the supported authentication methods:

• gcloud CLI: The most common approach for local development. When you log in with the gcloud CLI, Terraform can use those credentials automatically.
• Environment Variables: Terraform can use credentials set in environment variables like GOOGLE_APPLICATION_CREDENTIALS which points to a service account key file.
• Service Accounts: The most secure method for resources running in GCP. A Compute Engine instance or Cloud Build job can use a service account with the necessary permissions to access the GCS bucket.
• Workload Identity: The recommended way to authenticate for applications running on Google Cloud, especially with GKE, as it links a Kubernetes service account to a Google Cloud service account.

Examples

gcloud CLI

This is the easiest way to get started with local development. When you authenticate with the gcloud CLI, it sets up Application Default Credentials (ADC) which Terraform uses automatically.

Run Terraform commands without any extra authentication configuration:

terraform init

Set your project (if you have multiple):

gcloud config set project <your-project-id>

Log in to GCP:

gcloud auth application-default login

Environment Variables

This approach keeps sensitive information out of your code. You can download a service account key file and point to it using an environment variable.

export GOOGLE_APPLICATION_CREDENTIALS="/path/to/your/keyfile.json"

terraform init

Service Accounts

This is the most secure method for authenticating resources running inside GCP, such as a Compute Engine instance or Cloud Build job. You attach a service account to the resource, and Terraform assumes that service account's permissions automatically.

1. Create a Service Account with a role that grants permissions to access the GCS bucket. The Storage Object Admin role is a common choice.
2. Attach the Service Account to your Compute Engine instance or Cloud Build project. Terraform, when run on that resource, will automatically authenticate using the credentials provided by the service account. No secrets are stored on the resource itself.

---

Advanced Concepts

Partial Backend Configuration

Terraform won't let you use variables directly inside the backend block. But you can leave out sensitive or environment-specific values and supply them at runtime with a backend configuration file or command-line flags on terraform init.

Example using a file:

Run terraform init:

terraform init -backend-config="backend.conf"

backend.conf (sensitive info):

bucket = "my-terraform-state-bucket"

main.tf (partial config):

terraform {
  backend "gcs" {
    prefix = "my-app"
  }
}

This method prevents sensitive information from being committed to source control.

Managing Multiple Environments with Workspaces

If one configuration deploys to several environments, you can use Terraform workspaces to keep a separate state file for each one inside the same bucket.

Example:

# Create and switch to a development workspace
terraform workspace new dev

# Create and switch to a production workspace
terraform workspace new prod

When you switch between workspaces, Terraform automatically changes the prefix to include the workspace name (e.g., my-app/env:/dev/terraform.tfstate), ensuring each environment has its own isolated state file.

---

A Note on OpenTofu and Dynamic Backends

The classic Terraform CLI still refuses to let you put variables in a backend block. OpenTofu, the fork of Terraform, took a different path here.

Starting with version 1.8, OpenTofu lets you use variables and local values inside the backend block. That answers one of the oldest feature requests in the Terraform community.

In practice that means you can write a more flexible, DRY (Don't Repeat Yourself) backend configuration, which helps most when you run the same setup across several environments.

Here's how a dynamic gcs backend block could look in OpenTofu:

variable "env" {
  type    = string
  default = "dev"
}

terraform {
  backend "gcs" {
    bucket = "my-terraform-state-${var.env}-bucket"
    prefix = "my-app-${var.env}"
  }
}

Now you switch environments by changing the env variable. You can pass it at the command line:

tofu init -var="env=prod"

That cuts out the separate backend config files or wrapper scripts you'd otherwise write to juggle environments, so there's less to keep in sync by hand. If you manage a lot of near-identical environments, OpenTofu's dynamic backend blocks are worth a look.

This blog has been verified for Terraform and OpenTofu