Terraform Drift Detection for Cloud Native Infrastructure at Scale

Terraform Drift Detection in a Cloud Native Architecture

In a devops training workflow, Terraform makes infrastructure as code an auditable control plane for a cloud native architecture. Run a refresh-only plan on a schedule rather than a normal plan: it compares provider-read remote objects with the current state while deliberately ignoring configuration edits awaiting review. Exit code 0 means no remote drift, 2 means drift was found, and 1 is an execution failure that should page the platform team rather than create a drift ticket.

terraform init -input=false
terraform plan -refresh-only   -detailed-exitcode   -lock-timeout=5m   -out=refresh.tfplan
status=$?
terraform show -json refresh.tfplan > refresh-plan.json
exit $status

Use a locking remote backend before automating this check; otherwise two scheduled runners can refresh and write state concurrently, causing false findings or a lost serial update. For an S3-backed state, enable the native lock file and encryption, then grant the drift role only the bucket and cloud APIs it must read. A useful operational target is to measure drift_findings / managed_resources weekly and investigate any sustained increase instead of treating every changed timestamp as an incident.

terraform {
  backend "s3" {
    bucket       = "org-tf-state"
    key          = "platform/prod/terraform.tfstate"
    region       = "eu-west-1"
    encrypt      = true
    use_lockfile = true
  }
}

Add Drift Gates to a CI CD Pipeline

A ci cd pipeline should create a binary plan once, convert that exact plan to JSON for policy evaluation, and apply the saved binary plan only after approval. conftest evaluates Open Policy Agent rules against Terraform's machine-readable plan, which catches an exposed security-group rule before a provider API call. Do not use a text-plan grep: resource actions, unknown values, and nested blocks are represented structurally only in terraform show -json output.

terraform init -input=false
terraform plan -input=false -out=tfplan
terraform show -json tfplan > plan.json
conftest test plan.json --policy policy/
# Approved deployment job, using the same provider lock file:
terraform apply -input=false tfplan

Keep tfplan as a short-lived protected artifact, not a general build artifact. A saved plan can contain sensitive proposed values even when console output masks them; configure the CI system's restricted artifact retention and avoid uploading plan.json to public logs. Emit the counts from .resource_changes[].change.actions with jq and set an explicit review threshold, for example requiring platform approval for any delete action or more than 20 replacements. The subtle failure mode is applying a plan with a different provider binary or lock file than the planning job; commit .terraform.lock.hcl and run both jobs in the same pinned container image.

Kubernetes Training: Define Ownership for Container Orchestration

In kubernetes training, test ownership conflicts locally before introducing them into shared container orchestration clusters. Start minikube with the Docker driver, deploy a Terraform-managed workload, mutate it with kubectl, and inspect the next plan. This exercise also connects docker training to cluster behavior: the local Docker daemon runs the node, but Kubernetes controllers—not Docker—reconcile Deployment replicas and Pods.

minikube start --driver=docker
terraform apply -auto-approve
kubectl patch deployment api --type=merge   -p '{"spec":{"replicas":5}}'
terraform plan

Do not let Terraform and a HorizontalPodAutoscaler own spec.replicas simultaneously. The HPA continuously writes its calculated replica count; without an exception, every Terraform run plans to restore the static value and can defeat scaling during traffic. Ignore only the controller-owned path, while retaining drift detection for image, labels, service account, and pod security settings. Avoid ignore_changes = [metadata]: that broad exception can hide a manually added label used by a NetworkPolicy or admission controller.

resource "kubernetes_deployment" "api" {
  metadata { name = "api" }
  spec {
    replicas = 2
    selector { match_labels = { app = "api" } }
    template {
      metadata { labels = { app = "api" } }
      spec { container { name = "api" image = "registry.example/api@sha256:..." } }
    }
  }
  lifecycle {
    ignore_changes = [spec[0].replicas]
  }
}

AWS Training and Google Cloud Training: Keep Blast Radii Separate

For aws training and google cloud training, use explicit provider aliases and separate state roots for network, identity, and application layers. Aliases make a cross-cloud dependency visible in review instead of relying on whichever default credentials happen to be present on a runner. Pass providers into modules explicitly; this prevents a module intended for a production Google project from silently inheriting a default provider configured for a development project.

provider "aws" {
  alias  = "network"
  region = var.aws_region
}
provider "google" {
  alias   = "platform"
  project = var.gcp_project
  region  = var.gcp_region
}
module "vpc" {
  source    = "./modules/vpc"
  providers = { aws = aws.network }
}
module "gke" {
  source    = "./modules/gke"
  providers = { google = google.platform }
}

Authenticate CI with workload identity federation or cloud-native role assumption, not long-lived access keys stored as Terraform variables. Validate the identity at the start of each runner with aws sts get-caller-identity and gcloud auth list --filter=status:ACTIVE; fail if the expected account or project is absent. Keep a cross-cloud identifier such as a DNS zone name in a narrowly scoped output or secret store rather than reading an entire networking state: broad terraform_remote_state access exposes every output in that state, including outputs later marked sensitive.

Classify, Import, or Revert Terraform Drift

Treat a detected difference as one of three actions: revert an unauthorized change, codify an intentional change, or declare a controller-owned field. To codify an existing cloud object, add its resource configuration first and then use an import block; importing only the identifier does not generate safe configuration and the following plan may still propose destructive defaults. Review the post-import plan until it reaches zero changes before allowing normal applies.

import {
  to = aws_security_group.app
  id = "sg-0123456789abcdef0"
}

resource "aws_security_group" "app" {
  name        = "app"
  description = "Application ingress"
  lifecycle {
    prevent_destroy = true
  }
}

For unauthorized drift, record the resource address, provider identity, detected timestamp, and intended remediation in the incident ticket, then run a normal plan to verify that Terraform will restore the declared value. Use terraform state show aws_security_group.app to compare tracked attributes with provider reads before changing state manually; terraform state rm is not a repair command and can cause Terraform to attempt recreation. Track median time-to-remediate separately for access-control resources and cosmetic metadata, because a changed IAM binding has a materially different risk window from a changed tag.

Frequently Asked Questions

How do I run Terraform drift detection in a CI CD pipeline?

Schedule terraform plan -refresh-only -detailed-exitcode with a locked remote backend. Treat exit code 2 as a drift finding, upload a protected JSON rendering from terraform show -json for review, and reserve normal terraform plan for proposed configuration changes.

Can minikube help with Kubernetes training for Terraform-managed workloads?

Yes. Run minikube start --driver=docker, apply a small Terraform-managed Deployment, then patch its replicas with kubectl. The next plan demonstrates the Terraform-versus-HPA ownership conflict; configure ignore_changes only for controller-owned replica counts.

What should AWS training and Google Cloud training teach about Terraform state?

Use separate state roots per blast radius, explicit AWS and Google provider aliases, and federated CI identities. Verify runner identity with aws sts get-caller-identity and gcloud auth list; do not grant application pipelines read access to a shared networking state just to obtain one output.