January 28, 2019·4 min read·engineeringterraformstatelocking

Terraform state commands acquire locks in 0.11.12 beta

The January beta brings state editing into the locking protocol and separates remote state storage from remote execution.

A command that changes Terraform state participates in the same coordination problem as an apply. It can overwrite another writer's work even when it makes no API request to an infrastructure provider. Terraform's 0.11.12-beta1 release, published on January 28, addresses that problem by adding locking to state update commands. The same beta lets the remote backend support organizations that use hosted state without hosted operations.

These changes concern the division of responsibility between the command line and the backend. A shared state service needs clients that coordinate writes. A remote backend also needs to distinguish where it stores state from where it executes a plan. Treating either distinction as an implementation detail makes the operator's model of a command less reliable.

I value Terraform's ability to put a reusable configuration language over infrastructure APIs. State is part of the cost of that abstraction. Administrative commands deserve particular attention because they change the information Terraform uses to interpret everything that follows.

Locking before the state refresh

The implementation in pull request 19939 changes state mv, state push, and state rm. Each command gains a -lock flag that defaults to true and a -lock-timeout duration that defaults to zero. Their help text qualifies the guarantee. Terraform locks when the state implementation supports locking.

The position of the new code matters. The commands acquire the lock before refreshing the state they will modify. They arrange an unlock when the operation exits. This establishes a protected interval around reading the current snapshot and writing the modification. Merely locking around the final write would leave an opportunity to calculate a modification from an outdated snapshot.

Consider two cooperating clients that both read the same initial state. One removes an address while the other moves a different address. If each later writes a complete snapshot based on its original read, the later write can discard the earlier modification. This is a concurrency example derived from the read and write structure, not a report of an incident. Locking before the refresh prevents that particular interleaving when both clients use a backend with working locks.

The new options make the behavior visible at the command line. For a state move, the command can wait for a lock instead of failing immediately when another operation holds it.

terraform state mv -lock-timeout=30s aws_instance.old aws_instance.new

The resource addresses here are illustrative. The command changes their association in state, and the added timeout controls how long lock acquisition retries. It does not change the meaning of the address move or turn a state edit into an infrastructure deployment. The patch adds the same locking controls to the other two state update commands.

Moving between state files needs coordination on both sides. The patch locks the source and, when the command uses a separate destination state, also locks that destination before refreshing it. A lock on only the source would leave the destination exposed to a competing update. Separate acquisition also means this code should not be read as a promise of an atomic transaction across arbitrary state stores.

Hosted state with local operations

The other substantial change appears in pull request 20007. The remote backend reads the organization's entitlements and checks whether operations are available. It constructs a local backend that uses the remote backend for state. When the organization lacks the operations entitlement, that local implementation handles execution.

This makes the backend name less predictive of execution location. Remote state describes where Terraform stores its shared record. Remote execution describes where Terraform runs the operation. The patch supplies an implementation for the combination of hosted state and local execution instead of requiring those choices to move together.

That distinction affects credentials and connectivity. In the locally executed case, the machine running Terraform remains the place where provider processes perform their work. Choosing hosted state does not by itself move that work into a hosted runner. This follows from the local execution path introduced in the patch, and it is an important part of understanding the new entitlement behavior.

The versioned remote backend documentation shows the existing organization and workspace configuration. A single workspace configuration has this shape.

terraform {
  backend "remote" {
    organization = "company"
    workspaces {
      name = "my-app-prod"
    }
  }
}

The organization and workspace names are placeholders. Its documentation recommends keeping the token in the CLI credentials configuration instead of placing it in the backend block. The entitlement check supplies the execution decision, so the block does not need a second state-only backend type.

A preview with a narrower claim

I find the separation useful because shared state is valuable even when local execution remains appropriate. It gives the infrastructure abstraction a more precise operational meaning. The command still runs somewhere specific, and the state still needs a defined owner and a coordination protocol.

The release also fixes remote backend version constraint checking and comparison of versions with prerelease suffixes, as recorded in its release notes. Those changes matter to a beta whose users are explicitly running a prerelease client against a service. They address compatibility handling without implying that every client and service version can work together.

The documentation at this tag continues to describe the remote backend as a preview and advises against production workloads. That qualification belongs alongside the new support. The concrete advance in this beta is that state update commands join the locking mechanism, while the remote backend can select local execution for an organization that only has state storage.