Deep State: Create Terraform Code from the State File

The Deep

HashiCorp Terraform is a great and useful tool that every DevOps Practitioner should know for creating and maintaining Infrastructure as Code.

We use Terraform extensively here at BigPanda to fully manage and define our SAAS Infrastructure as Code.

One day, when testing different ways to create a large resource as code, I accidentally deleted a complex resource that I had created manually.

I needed to find an easy way to recreate the deleted resource, even though I did not have any local Terraform code definition file for it yet.

What I found led me to deep dive into Terraform’s state file, and I found a solution which allowed me to create a full Terraform HCL (HashiCorp Configuration Language) file 750 lines of code long, using just a few simple Terraform commands.

It was a valuable learning experience for me to better understand Terraform and how the state works under the hood and I hope this may help you as well.

The Challenge

The resource I deleted was created manually, and I needed to find a way to easily recreate it without having to write it all over again from scratch.

Luckily in my case, although I had deleted the resource remotely on the cloud provider, I had imported the resource into my local Terraform state.

Using careful Terraform state manipulation, I managed to create a full Terraform definition file from scratch and successfully re-created the deleted resource using a few simple Terraform commands which I’m going to share with you.

But first, let’s cover some basic Terraform concepts.

Provider

Terraform uses a provider architecture so that when working with a configuration, you must define which Provider you are working with.

A Terraform Provider is essentialy a Terraform code module that can be used to create resources of a Cloud Infrastructure or SAAS service as code.

Terraform uses that module definitions and parameters to know how to create and update your resources in the chosen Provider.

For example, when working with the Amazon Web Services Provider, you must specify the AWS Region you are working in and make sure to authenticate using IAM Roles or AWS Access and Secret Keys.

Resource

Terraform uses the providers to create different objects as code, they can be an AWS VPC, an S3 Bucket, or an RDS Instance.

When writing Terraform code, this is typically referred to as follows:

resource "resource_type" "resource_name" {
<RESOURCE_PARAMETERS_AND_CONFIGURATION>
}

State

The most important file to know when working with Terraform is the Terraform State file.

The state file, as its name suggests, is what the terraform binary interacts with to be able to know what the current state of the remote infrastructure is versus the desired state configured in the Terraform configuration language code.

The state is persisted between runs of Terraform as a physical file on local or remote storage, and the file is typically called terraform.tfstate

Before each run, Terraform will start by checking the known state, and then compares the Terraform definition files to the remote infrastructure environment.

Let’s dive into an example terraform.tfstate file.

{
"version": 4, # Version of Terraform state file syntax
"terraform_version": "1.1.2", # Version of Terraform binary being used
"serial": 31, # Serial number of this version of the state
"lineage": "4978360c-2654-49fc-38d2-c86f4dc0abfe", # Unique state ID
"outputs": {}, # Outputted resources
"resources": [
{
<ACTUAL RESOURCES LISTED HERE>
}
]
}

What we can see is that there is quite a lot of useful information available to us in the state file.

That information can be inspected manually to check what Terraform currently knows about the current state of the infrastructure.

There is even a command to display the full contents of the current state file:

terraform state pull

We can also list the resources managed by the current state file by using the command:

terraform state list

It is also possible to show a specific resource by specifying the resource ID as follows:

terraform state show resource_type.resource_name

Importing Remote State

Once we have resources in the cloud, even if there are no Terraform files locally that describe them as code, we can import them into our local state file using the terraform import command.

terraform import resource_type.resource_name unique_resource_id

Once the resources are imported into the Terraform state, they will appear in the state file and can be referenced from new Terraform code files or from the terraform binary.

Solution

Now that we know a bit more about Terraform state, let's try to combine the commands we learned to recreate our deleted resource!

As mentioned before, although the remote resource had been deleted, I still had the resource definition imported into my local state file. So I knew the resource definition was there and I just needed some state manipulation commands to extract the definition from the state.

Steps

1. Created an empty resource block in a Terraform main.tf file for representing the remote resource.
In my case, this was a datadog_synthetics_test resource.

resource "datadog_synthetics_test" "my_test" {

}

2. Imported the remote resource into local Terraform state

terraform import datadog_synthetics_test.my_test <RESOURCE_ID>

3. Exported the resource from the state into a new file my_test.tf

terraform state show -no-color datadog_synthetics_test.my_test > my_test.tf

NOTE: I used the -no-color flag, to have the target file as clean as possible without any weird shell color characters.

4. Removed IDs from the file created.

In my case with datadog_synthetics_test resource, the resource IDs are the Terraform resource id and the DataDog monitor id.
(Alternatively, you can just comment them out in the output file)

resource "datadog_synthetics_test" "my_test" {
# id = "***-***-***"
# monitor_id = ******
.
.
.
}

5. Applied the changes

terraform apply my_test.tf

6. Voilà!
My datadog_synthetics_test resource was re-created successfully!

Caveats

As mentioned, the steps above worked for me with the datadog_synthetics_test resource.
The versions of Terraform and the DataDog provider I used were:

❯ terraform -version
Terraform v1.1.2
+ provider registry.terraform.io/datadog/datadog v3.12.0

I have not extensively tested this method, and this will probably not be able to easily recreate all types of resources for all providers.

But I hope this solution can help you with some resources and providers.

Conclusion

Terraform is usually used to define infrastructure proactively from code to cloud, but this reverse Terraform solution can help us restore and recreate resources from cloud to code.

It’s useful to understand the Terraform state file, its format, and its contents when using Terraform for managing infrastructure as code.

This helps when resources are deleted by mistake or as a shortcut for creating valid Terraform code from existing resources using just a few simple Terraform commands.

Would love to hear if you find this useful and if this helped you as well!

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store