Configuration drift

On this page 7

Configuration drift is the divergence between the intended state of infrastructure, as declared in code or a configuration baseline, and the actual state of the running resources. In infrastructure as code, drift means that the cloud account, the servers or the network no longer match what the repository says they should be.

Drift matters because infrastructure as code is only trustworthy when the code is the single source of truth. Once production differs from the declared configuration, every plan, review and recovery procedure rests on a false picture, and the next automated change may silently revert a fix or break a dependency that nobody wrote down.

How configuration drift arises

Drift arises whenever a resource is changed outside the declared workflow. The most common cause is a manual change in a cloud console or command-line session, often made during an incident to restore service quickly and never written back into the code.

Not every change has a human author. Provider-side defaults change when a cloud platform adds a new attribute, autoscaling and self-healing systems replace instances, and other automation tools, such as a second pipeline or a security agent, modify the same resource. In Cloud environments with many teams, overlapping ownership of one resource is a frequent and underestimated source.

The following table groups the typical sources of drift by how they appear and how hard they are to notice.

SourceTypical triggerDetectability
Manual hotfixConsole edit during an incidentHigh if a refresh runs regularly
Out-of-band automationScript or second tool edits the resourceMedium, changes look legitimate
Provider default changeNew attribute or changed API behaviorLow, appears after a provider upgrade
Runtime mutationPackage updates or edited files on a serverLow without host-level scanning
Partial applyPipeline fails halfway through a changeMedium, state and reality disagree
Common sources of configuration drift

Drift in state files and live resources

Declarative tools such as Terraform and OpenTofu detect drift by comparing three views of the infrastructure: the configuration in code, the recorded state file, and the live resource as returned by the provider API. A refresh reads the live attributes, and a plan reports the differences that would be needed to bring reality back to the code.

Commands such as terraform plan -refresh-only show drift without proposing changes, which is the safest way to inspect it. Only attributes that the tool manages are compared, so a resource created entirely by hand stays invisible to the plan; finding such unmanaged resources requires an inventory scan of the account.

Procedural and agent-based tools see drift differently. Ansible checks the host at run time, and idempotent modules report a changed status when they correct something. Pull-based agents converge a host on a schedule, which corrects drift continuously but may also hide how often it occurs.

Tool styleDetection methodDefault reaction
Declarative provisioningRefresh and plan against stateReports a diff, waits for apply
Push configuration managementCheck mode on each runCorrects on the next run
Pull agentPeriodic local evaluationCorrects automatically
GitOps controllerContinuous comparison with a repositoryReconciles, often within minutes
How tool styles detect and handle drift

Configuration drift is distinct from version skew, state corruption and intentional divergence, although all four produce a mismatch. Separating them prevents the wrong remedy, for example overwriting a deliberate regional exception or importing a corrupted record.

Declarative configuration is the strongest structural defense, because a declared desired state makes every deviation measurable. Immutable infrastructure goes further and avoids in-place changes altogether: servers are replaced instead of patched, so runtime drift on a host has little time to accumulate.

ConceptWhat differsTypical remedy
Configuration driftLive resource versus declared codeReconcile or codify the change
Version skewTool, module or provider versionsPin and upgrade versions together
State corruptionState file versus reality, not codeRepair state, import or remove entries
Intentional divergenceDocumented exception per environmentExpress it as a variable or override

Detecting and remediating drift

Effective drift management combines scheduled detection, a clear decision rule and restricted write access. A scheduled pipeline that runs a refresh-only plan every few hours to once per day is common; production accounts with strict compliance needs often check at least daily, while sandbox accounts may not need checks at all.

Every detected difference requires a decision: revert the live resource to match the code, or codify the change because it was correct. Reverting blindly can undo an emergency fix; codifying blindly turns accidents into policy. The table summarizes a practical decision rule.

SituationRecommended actionReason
Unapproved console changeRevert through applyRestores the reviewed baseline
Valid emergency hotfixCodify, then applyKeeps the fix and the audit trail
Provider default addedDeclare or ignore the attributeStops recurring noise in plans
Resource created by handImport or deleteBrings it under management or removes it
Tool fighting toolAssign a single ownerEnds the endless change loop
Decision table for detected drift

Prevention reduces the volume of drift more than detection does. Read-only console access for most engineers, change through pull requests only, cloud audit logs that flag writes from human identities, and policy-as-code guardrails keep the declared workflow the only path to production.

Pitfalls in managing configuration drift

The most damaging mistake is treating drift alerts as noise. When plans constantly show harmless differences, engineers stop reading them, and a meaningful change, such as an opened firewall port, passes unnoticed.

  • Overusing ignore rules, such as lifecycle ignore settings, hides real drift along with the noise.
  • Auto-remediating everything in production can revert an incident fix while the incident is still open.
  • Detecting only managed attributes misses unmanaged resources, which are a common audit finding.
  • Long-lived environments without checks accumulate drift until a rebuild from code fails or differs.
  • Shared resources edited by two tools produce oscillating plans that never converge.

Drift also has a security dimension. An unexpected change to an identity policy, a security group or an encryption setting may be the first sign of a compromised credential, so drift reports for security-relevant resources deserve the same attention as monitoring alerts.

The simple explanation

Imagine a building with an official floor plan. Over the years, someone moves a wall, another person adds a door, and nobody updates the plan. The plan still looks correct, but it no longer describes the real building.

Configuration drift is the same problem for computer systems. Teams write down how their servers and networks should look, and then small undocumented changes pile up. Catching drift early keeps the written plan honest, so the systems can be rebuilt or changed safely.

How it is used in practice

A platform team manages about 400 cloud resources across three accounts with Terraform. During a late-night outage, an engineer raises a database connection limit and opens a security group rule in the console. Service recovers, and the change is never written into the code.

A nightly refresh-only plan reports both differences the next morning. The team reviews them in the daily standup: the connection limit was a valid fix and is codified in a pull request, while the broad security group rule is reverted through a normal apply. Without the scheduled check, the next routine deployment would have reset the connection limit and triggered the same outage again.