I opened the AWS console, selected an EC2 instance, added a tag, and saved it.

The tag did not exist in Terraform. The deployed infrastructure now said one thing while the code said another.

That was the test.

My drift detector already knew how to find the difference and classify the EC2 update as LOW. The missing part was closing the loop: once the system understood that the change was low severity, it needed to restore the infrastructure without waiting for me to run Terraform manually.

So I added a separate remediation pipeline.

The Stack

Manual EC2 tag added in AWS
        ↓
Drift CodeBuild runs terraform plan
        ↓
SNS publishes the structured change
        ↓
Lambda classifies the update as LOW
        ↓
Remediation CodeBuild runs terraform apply
        ↓
Terraform removes the unmanaged tag

Detection and remediation use different CodeBuild projects. The detector reads the infrastructure and reports differences. The remediation project has a separate IAM role because it can change AWS resources.

Step 1: Make EC2 Updates Eligible for Remediation

The classifier already grouped resource types into HIGH, MEDIUM, and LOW categories.

An EC2 instance belongs to MEDIUM_RISK_TYPES. Initially, that meant even a tag update was classified as MEDIUM and stopped before remediation.

The resource type alone was not enough. The action mattered too.

I updated the classifier so an in-place update on a MEDIUM resource becomes LOW, while creates remain MEDIUM and delete or replacement actions become HIGH:

def classify_change(resource_type, actions):
    if resource_type in HIGH_RISK_TYPES:
        return "HIGH"
    elif resource_type in MEDIUM_RISK_TYPES:
        if "delete" in actions or "replace" in actions:
            return "HIGH"
        elif "update" in actions:
            return "LOW"
        return "MEDIUM"
    else:
        return "LOW"

For this test, Terraform reported the manually added EC2 tag as an update. That moved it into the LOW path.

Step 2: Keep Deletions Out of the Automatic Path

LOW did not automatically mean “start Terraform.” I also filtered out deletions:

low_changes_to_remediate = [
    change for change in classified["LOW"]
    if "delete" not in change.get("actions", [])
]

This distinction matters. Updating an existing resource to match the code is different from recreating something that a person deliberately removed. Deletions stay in the manual-review path.

If an eligible LOW change remains, Lambda starts the remediation CodeBuild project:

if low_changes_to_remediate:
    codebuild.start_build(
        projectName=os.environ.get(
            "REMEDIATION_PROJECT_NAME",
            "terraform-drift-remediation",
        )
    )

The Lambda role only needs permission to start that specific project:

{
  Effect   = "Allow"
  Action   = ["codebuild:StartBuild"]
  Resource = aws_codebuild_project.remediation.arn
}

Lambda makes the routing decision. CodeBuild performs the repair.

Step 3: Run Remediation Separately

I did not add terraform apply to the existing drift detector. Detection should remain read-only even when remediation fails or is disabled.

The second build starts in a clean environment, installs Terraform, clones the Three-Tier repository, and applies the declared configuration:

version: 0.2

phases:
  install:
    commands:
      - curl -o terraform.zip https://releases.hashicorp.com/terraform/1.10.0/terraform_1.10.0_linux_amd64.zip
      - unzip terraform.zip -d /usr/local/bin
  pre_build:
    commands:
      - git clone https://github.com/lalitbagga/Three-Tier-Infra.git /tmp/Three-Tier-Infra
      - cd /tmp/Three-Tier-Infra
      - terraform init
  build:
    commands:
      - cd /tmp/Three-Tier-Infra
      - terraform apply -auto-approve -lock=false

The remediation project uses its own CodeBuild role with write access to the AWS services managed by the Three-Tier project. During implementation, Terraform exposed several missing permissions as it evaluated and changed resources, so I expanded the experimental role to cover the project.

That role is intentionally separate from the detector. A job that only runs terraform plan should not inherit the permissions required for terraform apply.

CodeBuild Was Using Old Terraform Code

The first remediation run included changes I was not expecting. CodeBuild clones the Terraform project from GitHub, but my latest changes existed only on my laptop.

I pushed the current code and ran the test again. This time, Terraform removed only the EC2 tag as expected.

In CI, the committed repository, not the code on your laptop, is the configuration Terraform will enforce.

The Tag Disappeared

With the current repository in GitHub, I repeated the test:

  1. Opened the EC2 instance in the AWS console.

  2. Added a tag manually and saved it.

  3. Started the Terraform drift detector.

  4. The plan reported an update to the EC2 instance.

  5. Lambda classified the update as LOW.

  6. Lambda started the remediation CodeBuild project.

  7. CodeBuild ran Terraform apply.

  8. I returned to the EC2 Tags view and confirmed that the manually added tag was gone.

Terraform restored the infrastructure to what was declared in Git.

Git configuration: tag absent
AWS resource:       tag added manually
Terraform result:   tag removed

The system was no longer only telling me about drift. For the LOW update I tested, it detected the change, classified it, and repaired it.

MEDIUM, HIGH, and deletion paths still stop for human review. I will add that approval workflow later in the project. For now, automatic remediation only handles LOW-severity changes that the system considers safe to correct without human intervention.

What Comes Next

The infrastructure is back in the expected state, but the evidence is spread across CodeBuild and CloudWatch logs.

The next part of the system will keep a remediation history in a database, expose it through an API, and visualize it in Grafana. I want to see what changed, how it was classified, whether remediation ran, and whether it succeeded without reconstructing the story from separate AWS services.

Detection found the tag. Classification decided it was LOW. Remediation removed it. Now the system needs to remember that it happened.