How To Backup Terraform State in a Remote Backend

It is very common to want to back up your Terraform or OpenTofu state when using a remote operations backend like Scalr or Terraform Cloud. When the Terraform runs are executed in Scalr and the state file is then stored, it is stored encrypted either in the Scalr cloud backend or in a bucket service of your choice. In both scenarios, the state is encrypted and the application has the key to unencrypt it. In this case, you may want to have an option to back up the state in the location of your choice for some of your more mission-critical infrastructure.

In this blog, we’ll talk about the various ways to back up your state file from a backend like Terraform Cloud or Scalr. In this case, we’ll be using Scalr.

In any of the scenarios that we talk about below you will need a Scalr token, so let’s start with that. To authenticate to the API, you can either use a personal access token or a service account token. A personal token might be good for a one-off scenario, but if you’re doing regular backups, then you’ll want a service account token.

Personal access tokens can be obtained by logging into Scalr and clicking on your profile on the bottom left of the screen:

‍

You can then generate and manage them from your token page:

Service account tokens can be obtained by going to the IAM section in Scalr and clicking on service accounts:

One Off Terraform State Pull

The first use case is pulling a specific state file down in a one-off scenario. To start, you must add the Terraform remote backend block to your Terraform configuration files:

terraform {
  backend "remote" {
    hostname = "<my-account>.scalr.io"
    organization = "<ID of environment>"
    workspaces {
      name = "<workspace-name>"
    }
  }
}

Double check that you have updated the settings in the remote backend configuration to ensure you have the correct workspace name and environment set.

Once confirmed, run the following, depending on if you are using Terraform or OpenTofu, which will validate you can connect to the backend:

terraform init
tofu init

Once confirmed that the backend is connected, pull the state down with the standard Terraform or OpenTofu CLI:

terraform state pull> terraform.tfstate
tofu state pull> tofu.tfstate

That’s it! You now have the state pulled down to do whatever you choose to do with it.

State Bulk Backups

In the next use case, we’ll talk about how to do bulk backups of all of your state files. Some organizations might want to do this on a regular cadence and an external script can be written to pull the state files down nightly for example. We have written a python script, which can be found here that will do exactly that.

To start, you’ll want to install the pip requirements:

pip3 install -r requirements.txt

Then simply run the following command to pull all state files:

download_state_files.py --host example.scalr.io --token xxx --output-dir /tmp/scalr-state-files

Remember to update the command above and add your Scalr account as well as the token.

Now you can choose to run this manually or to set up an automated cadence to pull down the state files.

Backup After a Terraform Apply

The last use case is backing up the Terraform state files in basically real-time. In this case, we’ll use Scalr custom hooks that run before and after Terraform init, Terraform plan, and Terraform apply. The custom hooks functionality allows you to customize the workflow or export objects such as the Terraform state file. 

You’ll find the custom hooks option in the workspace settings:

To back up the state after a Terraform apply, simply add a command to pull the state and copy it into the location of your choice. In this case, we’ll pull the state file and add it to our own S3 bucket using the standard AWS CLI commands:

aws s3 cp <(terraform show) s3://my-bucket/my-state.json

Scalr has the command line tools automatically installed for AWS, Azure, and GCP. 

Before executing a run, you must ensure that you are passing your AWS credentials as environment variables into the run. This can be done by exporting the provider configurations that you have set in Scalr or by adding your own in the workspace:

Now after every apply you’ll see the custom hook run and the Terraform state file sent to the S3 bucket.