How To Update Terraform State In A Remote Backend

Intro 

It is very common to have the need to import, update, or manipulate a Terraform state file as you’re working through Terraform deployments. Sometimes this needs to be done because a mistake was made in the current state ,you just need to make a quick update without having to completely refactor your code or redeploy, or you have a new Terraform project involving migrations. This is straightforward when using a local state file, but today we’ll talk about some of the methods that Terraform supports when using a remote state file in a remote backend.

In all of the scenarios listed below, you will need to ensure that you have the remote backend setup first, which could be something like Terraform Cloud or Scalr. In this case, we’ll be using Scalr:

First, get the API token for Scalr by running :

terraform login <account-name>.scalr.io

This will redirect you to scalr.io to retrieve the API token:

Terraform will request an API token for scalr-demo.scalr.io using your browser.

If login is successful, Terraform will store the token in plain text in
the following file for use by subsequent commands:
    /Users/demo/.terraform.d/credentials.tfrc.json

Do you want to proceed?
  Only 'yes' will be accepted to confirm.

  Enter a value: yes


---------------------------------------------------------------------------------

Terraform must now open a web browser to the tokens page for scalr-demo.scalr.io.

If a browser does not open this automatically, open the following URL to proceed:
    https://demo.scalr.io/app/settings/tokens?source=terraform-login


---------------------------------------------------------------------------------

Generate a token using your browser, and copy-paste it into this prompt.

Terraform will store the token in plain text in the following file
for use by subsequent commands:
    /Users/demo/.terraform.d/credentials.tfrc.json

Token for demo.scalr.io:
  Enter a value:


Retrieved token for user demo@scalr.com

Add the remote backend settings to your Terraform config files:

terraform {
  backend "remote" {
    hostname = "my-account.scalr.io"
    organization = "<ID of environment>"
    workspaces {
      name = "<workspace-name>"
    }
  }
}

These backend settings will be different per workspace and environment, be sure to update this as you go.

That’s all you need to do to get started, going forward any Terraform commands will run remotely in Scalr.

Migrate State

The first use case is migrating Terraform state file into the remote backend. There are two scenarios for migrating state, from local to remote or remote to remote.

For local to remote, you simply need to add the remote block as seen in the intro. For remote to remote, just update your remote backend settings to point to your new remote backend. For example, moving from S3 to Scalr:

terraform {
 backend "s3" {
   bucket = "bucket"
   key    = "path/to/key"
   region = "us-west-1"
 }
}

Now becomes:

terraform {
  backend "remote" {
    hostname = "my-account.scalr.io"
    organization = "<ID of environment>"
    workspaces {
      name = "<workspace-name>"
    }
  }
}

If you are migrating state, this means you likely have the state being created from some other Terraform workflow already. It is a best practice to stop any jobs that might update the state while you’re doing the migration.

Double check that you have updated the settings in the remote backend configuration to ensure you have the correct workspace name and environment set. 

Once confirmed, run the following:

terraform init

Output to expect:

Initializing the backend...
Backend configuration changed!

Terraform has detected that the configuration specified for the backend
has changed. Terraform will now check for existing state in the backends.


Terraform detected that the backend type changed from "s3" to "remote".
Do you want to copy existing state to the new backend?
  Pre-existing state was found while migrating the previous "s3" backend to the
  newly configured "remote" backend. No existing state was found in the newly
  configured "remote" backend. Do you want to copy this state to the new "remote"
  backend? Enter "yes" to copy and "no" to start with an empty state.

  Enter a value: yes


Successfully configured the backend "remote"! Terraform will automatically
use this backend unless the backend configuration changes.

Initializing provider plugins...

The following providers do not have any version constraints in configuration,
so the latest version was installed.

To prevent automatic upgrades to new major versions that may contain breaking
changes, it is recommended to add version = "..." constraints to the
corresponding provider blocks in configuration, with the constraint strings
suggested below.

* provider.aws: version = "~> 2.70"

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

That’s it! You might get prompted to set variables that do not have a value, but at this point the state is imported and the migration is done.

Update State

Another common use case is that you want to manually update the state that already exists in Scalr and ensure the updated state is pushed back into Scalr.

First, pull down the state locally. If you are using a VCS based workspace, update it to CLI based, add the remote configuration to your local configuration (see intro), and pull it down locally:

terraform state pull> terraform.tfstate

Now, you can make the updates to your terraform.tfstate file. But, the key here is that the serial of the local state must be at least one number higher than the serial in the remote state. For example, if the state in Scalr shows the following (serial set to 3):

Then ensure that the new state locally has serial set to 4 before pushing:

{
  "version": 4,
  "terraform_version": "0.13.5",
  "serial": 4,
…
..
.

If not, you will see the following error: “Failed to write state: cannot overwrite existing state with serial 1 with a different state that has the same serial."

Once the updates are made and the serial is correct, push the state in:

terraform state push terraform.tfstate

Output to expect:

Acquiring state lock. This may take a few moments...

Releasing state lock. This may take a few moments...

Import Resources

The use case to import resources into remote backend state is very straight forward as this works exactly as you would expect it to without the remote backend. All you have to do is define the resource and ADDRESS ID. For example, I want to import an AWS instance (i-0d13d5591a41e18ef) to the resource aws_instance.scalr. This is done by running the following command:

terraform import aws_instance.scalr i-0d13d5591a41e18ef

And getting the following output:

aws_instance.scalr: Importing from ID "i-0d13d5591a41e18ef"...
aws_instance.scalr: Import prepared!
  Prepared aws_instance for import
aws_instance.scalr: Refreshing state... [id=i-0d13d5591a41e18ef]

Import successful!

The resources that were imported are shown above. These resources are now in
your Terraform state and will henceforth be managed by Terraform.

Releasing state lock. This may take a few moments...

Terraform Move

In Terraform 1.1.x, the moved block was introduced, which allows you to refactor the code without making breaking changes. The good news for remote backends is that it works as expected right out of the box.

In this example, the aws_instance block was updated to scalr1 from scalr:

resource "aws_instance" "scalr1" {
  ami                    = "ami-27571234"
  instance_type          = "t2.nano"
  subnet_id              = "subnet-0ebb1058ad721abcd"
  vpc_security_group_ids = ["sg-0880cfdc546b12345"]
  key_name               = "key"
}

The moved block was added to the code as well to notify Terraform of the change:

moved {
  from = aws_instance.scalr
  to   = aws_instance.scalr1
}

Changes were applied:

terraform apply

And getting the following output was produced:

Running apply in the remote backend. Output will stream here. Pressing Ctrl-C
will cancel the remote apply if it's still pending. If the apply started it
will stop streaming the logs, but will not stop the apply running remotely.

Preparing the remote apply...

To view this run in a browser, visit:
https://scalrdemo.scalr.io/app/org-ssccu6d5ch64345/myapp1/runs/run-trqa57jnfo21234

Waiting for the plan to start...

Terraform v1.1.3
Initializing Terraform configuration...
Configuring remote state backend...
Initializing plugins and modules...

------------------------------------------------------------------------

aws_instance.scalr3: Refreshing state... [id=i-07c3403aa72a17842]
aws_instance.scalr1: Refreshing state... [id=i-0d13d5591a41e18ef]

Terraform will perform the following actions:

  # aws_instance.scalr has moved to aws_instance.scalr1
    resource "aws_instance" "scalr1" {
        id                                   = "i-0d13d5591a41e18ef"
        tags                                 = {}
        # (29 unchanged attributes hidden)






        # (6 unchanged blocks hidden)
    }

Very straightforward and no tricks were needed to make it work with the backend.

Terraform Refresh-Only

The Terraform refresh-only command will update your state to match the actual state of the resources in the cloud. For example, someone may have updated the tags on a S3 bucket directly in the AWS console, but never updated the Terraform code or state. The refresh-only command will detect the drift and then ask the user if they want to update the Terraform state file to match:

Terraform v1.5.7
Initializing Terraform configuration...
Configuring remote state backend...
Initializing plugins and modules...
------------------------------------------------------------------------
aws_s3_bucket.this[0]: Refreshing state... [id=terraform-202410071900553970000]
aws_s3_bucket_public_access_block.this[0]: Refreshing state... [id=terraform-202410071900553970000]
Note:
Objects have changed outside of Terraform
Terraform detected the following changes made outside of Terraform since the
last "terraform apply" which may have affected this plan:
# aws_s3_bucket.this[0]
has changed
~
resource "aws_s3_bucket" "this" {
id = "terraform-202410071900553970000"
+
tags = {
+
"Name" = "NewTag"
}
~
tags_all = {
+
"Name" = "NewTag"
}
# (11 unchanged attributes hidden)
# (3 unchanged blocks hidden)
}
This is a refresh-only plan, so Terraform will not take any actions to undo these. If you were expecting these changes then you can apply this plan to record the updated values in the Terraform state without changing any remote objects.

Terraform Remove

In the event that resources have been destroyed directly in the cloud console or by something other mechanism, the terraform state rm command can be used to remove resources from state:

terraform state rm
At least one address is required.

Usage: terraform [global options] state rm [options] ADDRESS...

Summary

Those were a few of the best practices when needing to update a Terraform state file when using a remote backend. Interested in more topics like this? Check out this blog, which talks about topics like the Terraform refresh command, terraform imports, and the terraform mv command.