Managing Terraform State: Best Practices for DevOps

🔔This article, will explain in detail the understanding of Terraform state, its purpose, and how to store and manage state files across different environments🔔

❄️Synopsis:

🌿 Understand how Terraform state works, its significance in infrastructure as code, and how to manage the state effectively in a team environment.

---

❄️Terraform State:

🌿Every time you run Terraform, it records information about what infrastructure it created in a Terraform state file. By default, when you run Terraform in the folder /foo/bar, Terraform creates the file /foo/bar/terraform.tfstate. This file contains a custom JSON format that records a mapping from the Terraform resources in your configuration files to the representation of those resources in the real world.

The state file format is a private API that is meant only for internal use within Terraform. You should never edit the Terraform state files by hand or write code that reads them directly.

---

❄️Shared Storage for State Files:

🌿The most common technique for allowing multiple team members to access a common set of files is to put them in version control (e.g., Git). Although you should store your Terraform code in version control, storing Terraform state in version control is a bad idea for the following reasons:

Manual error - It’s too easy to forget to pull down the latest changes from version control before running Terraform or to push your latest changes to version control after running Terraform. It’s just a matter of time before someone on your team runs Terraform with out-of-date state files and, as a result, accidentally rolls back or duplicates previous deployments.

Locking - Most version control systems do not provide any form of locking that would prevent two team members from running terraform apply on the same state file at the same time.

Secrets - All data in Terraform state files is stored in plain text. This is a problem because certain Terraform resources need to store sensitive data. For example, if you use the aws_db_instance resource to create a database, Terraform will store the username and password for the database in a state file in plain text, and you shouldn’t store plain text secrets in version control

🌿A Terraform backend determines how Terraform loads and stores state. The default backend, which you’ve been using this entire time, is the local backend, which stores the state file on your local disk. Remote backends allow you to store the state file in a remote, shared store. A number of remote backends are supported, including Amazon S3, Azure Storage, Google Cloud Storage, and HashiCorp’s Terraform Cloud and Terraform Enterprise.

🌿If you’re using Terraform with AWS, Amazon S3 (Simple Storage Service), which is Amazon’s managed file store, is typically your best bet as a remote backend for the following reasons:

1. It’s a managed service, so you don’t need to deploy and manage extra infrastructure to use it.

2. It’s designed for 99.999999999% durability and 99.99% availability, which means you don’t need to worry too much about data loss or outages.

3. It supports encryption, which reduces worries about storing sensitive data in state files. You still have to be very careful who on your team can access the S3 bucket, but at least the data will be encrypted at rest (Amazon S3 supports server-side encryption using AES-256) and in transit (Terraform uses TLS when talking to Amazon S3).

4. It supports locking via DynamoDB.

5. It supports versioning, so every revision of your state file is stored, and you can roll back to an older version if something goes wrong.

6. It’s inexpensive, with most Terraform usage easily fitting into the AWS Free Tier.

🌿To enable remote state storage with Amazon S3, the first step is to create an S3 bucket.

provider "aws" {
  region = "us-east-2"
}

🌿create an S3 bucket by using the aws_s3_bucket resource

 resource "aws_s3_bucket" "terraform_state" {
  bucket = "terraform-up-and-running-state"
  # Prevent accidental deletion of this S3 bucket
 lifecycle {
    prevent_destroy = true
  }
 }

bucket This is the name of the S3 bucket. Note that S3 bucket names must be globally unique among all AWS customers

prevent_destroy When you set prevent_destroy to true on a resource, any attempt to delete that resource (e.g., by running terraform destroy) will cause Terraform to exit with an error.

🌿First, use the aws_s3_bucket_versioning resource to enable versioning on the S3 bucket so that every update to a file in the bucket actually creates a new version of that file.

 # Enable versioning so you can see the full revision history of your
 # state files
 resource "aws_s3_bucket_versioning" "enabled" {
  bucket = aws_s3_bucket.terraform_state.id
 versioning_configuration {
    status = "Enabled"
  }
 }

🌿Second, use the aws_s3_bucket_server_side_encryption_configuration resource to turn server-side encryption on by default for all data written to this S3 bucket.

 # Enable server-side encryption by default
 resource "aws_s3_bucket_server_side_encryption_configuration" "default" {
  bucket = aws_s3_bucket.terraform_state.id
 rule {
 apply_server_side_encryption_by_default {
      sse_algorithm = "AES256"
    }
  }
 }

🌿Third, use the aws_s3_bucket_public_access_block resource to block all public access to the S3 bucket. S3 buckets are private by default, but as they are often used to serve static content—e.g., images, fonts, CSS, JS, HTML—it is possible, even easy, to make the buckets public.

 # Explicitly block all public access to the S3 bucket
 resource "aws_s3_bucket_public_access_block" "public_access" {
  bucket = aws_s3_bucket.terraform_state.id                 
  block_public_acls = true      
  block_public_policy  = true    
  ignore_public_acls   = true
  restrict_public_buckets = true
 }

🌿Create a DynamoDB table to use for locking. DynamoDB is Amazon’s distributed key-value store. It supports strongly consistent reads and conditional writes, which are all the ingredients you need for a distributed lock system.To use DynamoDB for locking with Terraform, you must create a DynamoDB table that has a primary key called LockID

 resource "aws_dynamodb_table" "terraform_locks" {
  name = "terraform-up-and-running-locks"        
  billing_mode = "PAY_PER_REQUEST"
  hash_key = "LockID"
  attribute {
    name = "LockID"
    type = "S"
  }
 }

🌿After everything is deployed, you will have an S3 bucket and DynamoDB table, but your Terraform state will still be stored locally. To configure Terraform to store the state in your S3 bucket (with encryption and locking), you need to add a backend configuration to your Terraform code. This is a configuration for Terraform itself, so it resides within a Terraform block and has the following syntax:

 terraform {
 backend "<BACKEND_NAME>" {
 [CONFIG...]
  }
 }

🌿where BACKEND_NAME is the name of the backend you want to use (e.g., "s3") and CONFIG consists of one or more arguments that are specific to that backend (e.g., the name of the S3 bucket to use). Here’s what the backend configuration looks like for an S3 bucket:

 terraform {
 backend "s3" {
    # Replace this with your bucket name!
    bucket = "terraform-up-and-running-state"       
    key  = "global/s3/terraform.tfstate"           
    region  = "us-east-2"        

    # Replace this with your DynamoDB table name!
    dynamodb_table = "terraform-up-and-running-locks"
    encrypt = true        
  }
 }

🌿Terraform will automatically pull the latest state from this S3 bucket before running a command and automatically push the latest state to the S3 bucket after running a command.

output "s3_bucket_arn" {
  value = aws_s3_bucket.terraform_state.arn       
  description = "The ARN of the S3 bucket"
 }
 output "dynamodb_table_name" {
  value = aws_dynamodb_table.terraform_locks.name      
  description = "The name of the DynamoDB table"
 }

---

❄️Hands-on Demo:

🌿 Here the output of terraform

🌿 S3 Bucket

🌿 Dynamo DB

---

🕵🏻I also want to express that your feedback is always welcome. As I strive to provide accurate information and insights, I acknowledge that there’s always room for improvement. If you notice any mistakes or have suggestions for enhancement, I sincerely invite you to share them with me.

🤩 Thanks for being patient and following me. Keep supporting 🙏

Clap👏 if you liked the blog.

For more exercises — please follow me below ✅!

vjraghavanv.hashnode.dev

#aws #terraform #cloudcomputing #IaC #DevOps #tools #operations #30daytfchallenge #HUG #hashicorp #HUGYDE #IaC #developers #awsugmdu #awsugncr #automatewithraghavan