How To Manually Edit A Terraform Cloud State File

There are rare occasions where you will be required to manually edit a Terraform state file. One such scenario would be if you manually delete Azure resources through the Azure GUI and subsequent terraform apply commands also attempt to delete the already-deleted resources (and fails).

In this case, a Terraform Cloud state refresh will not pick up that the "missing" resources have already been deleted, and future terraform apply commands will continue to fail.

Your options are to either execute terraform state rm resource.name, or manually remove the resource from the Terraform state file.

Unfortunately, Terraform Cloud does not offer a native mechanism to edit state files, but we can still do it manually.

In short, the procedure involves:

1. Downloading the newest Terraform Cloud state file to your local machine
2. Locally manipulating the downloaded Terraform Cloud .tfstate file
3. Reconfiguring the Terraform Cloud Workspace's main.tf on your local machine to use the backend "remote" {} backend instead of the cloud {} backend
4. Initializing the Terraform code on your local machine
5. Pushing the manipulated Terraform Cloud state file back to Terraform Cloud
6. Restoring your original backend and re-initializing it, locally

Procedure

• Ensure you have a copy of the Terraform code downloaded to your machine

• On your workstation, delete the .terraform/ and terraform.tfstate.d/ folders if they exist (they will only exist if you've previously run terraform commands in the stack).

• In your main terraform block (I like to use providers.tf), temporarily comment out the cloud {} code block.

  terraform {
  #  cloud {
  #    hostname = "app.terraform.io"
  #    organization = "my_organization"
  #    workspaces {
  #      tags = ["my_app"]
  #    }
  #  }
  }

Manually editing Terraform Cloud state file

• Still in providers.tf, configure a backend "remote" {} code block inside the main terraform {} code block.

  terraform {
  #  cloud {
  #    hostname = "app.terraform.io"
  #    organization = "my_organization"
  #    workspaces {
  #      tags = ["my_app"]
  #    }
  #  }
   
    backend "remote" {
      hostname = "app.terraform.io"
      organization = "my_organization"
      workspaces {
        name = "my_workspace_name"
      }
    }
  }

• Execute on your machine: terraform init -reconfigure

• In Terraform Cloud, go to your Workspace -> State -> Download the latest state file and save it to /local/path/to/repo/

• Manipulate the downloaded state file as needed.

• Upload the modified .tfstate file to Terraform Cloud by executing: terraform state push /local/path/to/repo/ filename.tfstate

  Troubleshooting: If you get the error "Failed to write state: cannot overwrite existing state with serial number with a different state that has the same serial", simply increment the "serial" number at the top of the downloaded .tfstate file and retry terraform state push

• Remove the temporary backend "remote" {} code block from providers.tf and uncomment the original cloud {} code block:

  terraform {
    cloud {
      hostname = "app.terraform.io"
      organization = "my_organization"
      workspaces {
        tags = ["my_app"]
      }
    }
  }

• Execute on your machine: terraform init -reconfigure

References

• https://support.hashicorp.com/hc/en-us/articles/360001151948
• https://developer.hashicorp.com/terraform/language/settings/backends/remote