March 21, 2018·4 min read·engineeringterraformstatelocking

Terraform 0.11.5 closes a state lock cleanup gap

Four commands acquired state through a shared backend method without releasing its lock. Terraform 0.11.5 adds the missing cleanup.

A shared helper can change a command's responsibilities without changing the command's apparent purpose. Terraform's console and graph commands illustrate the problem in 0.11.5. They obtained state through a backend method that acquired a lock, but did not release it themselves. The March 21 release fixes the same omission in import and push. The interesting failure was the mismatch between shared initialization and command-specific cleanup.

The patch for pull request 17645, authored by James Bardin on March 20, explains the cause directly. Locking improvements to regular commands had also caused these four commands to lock state. Their calls to Backend.Context acquired a lock whenever possible, which meant callers needed to arrange an explicit unlock. The fix adds that missing responsibility to every command calling the method directly.

I consider this kind of lifecycle consistency central to Terraform's usefulness. A declarative interface reduces the amount of procedural work an operator writes, but the implementation still has to acquire and release shared resources correctly. A state lock left behind by an inspection command interrupts the next operation just as effectively as a lock left behind by an apply.

Acquisition and release crossed different boundaries

The patch adds deferred cleanup in each affected command after context creation succeeds. Its repeated structure is short enough to examine without reconstructing the rest of Terraform's command framework.

defer func() {
    err := opReq.StateLocker.Unlock(nil)
    if err != nil {
        c.Ui.Error(err.Error())
    }
}()

This is code from Terraform's implementation, not configuration for a user to add. The command schedules unlocking for its return path. It also reports an unlocking error through the user interface. The cleanup therefore does not silently discard a failure at the end of an otherwise successful operation.

The design issue is visible in the placement of that code. Context creation does more than return an object that the caller can inspect. It acquires an external coordination resource. Any caller that knows only about the returned context can miss that side effect, particularly when the caller's main activity sounds observational. Console and graph are useful examples because their names do not suggest ownership of a state-writing lifecycle.

That does not imply Terraform should avoid locking them by assumption. The patch's stated solution preserves acquisition and supplies the corresponding release. It fixes an incomplete lifecycle without weakening coordination. The affected commands can continue using the shared initialization path, provided they complete the obligations that path introduces.

The regression test observes the remaining lock

The same patch extends an import test. After the command returns successfully, the test checks that the local state lock-information file does not exist. A remaining file fails the test before the rest of the assertions about provider configuration.

That observation matters because a successful exit alone did not establish successful cleanup. A command could complete its principal work and still leave the state unavailable to the next caller. The added assertion tests a consequence outside the immediate return value. It makes the lifecycle requirement explicit in the behavior that the test accepts.

The test is narrower than a proof about every backend. It checks the local lock artifact in that import scenario. Nevertheless, it targets the reported defect directly, and the production change applies the same deferred-unlock pattern to all four callers. This is a useful division between a concrete regression check and a shared implementation correction.

I prefer this evidence to treating the absence of an error message as proof that coordination worked. Resource cleanup often becomes visible only when the following operation tries to acquire the same resource. An assertion immediately after the command can catch that failure closer to its cause.

Why manual unlocking remains exceptional

The locking documentation in 0.11.5 explains the normal contract. When a backend supports locking, Terraform automatically locks operations that could write state. If acquisition fails, the operation stops. A quiet command does not mean locking was absent, since ordinary successful acquisition does not produce a message.

The documentation provides force-unlock for cases where automatic unlocking fails. It requires the unique lock identifier reported by Terraform. That identifier directs the request at a particular lock, but it does not establish that the process owning the lock has stopped. The documentation restricts the intended use to the caller's own lock after failed automatic cleanup.

Removing a lock held by an active operation can admit multiple writers. That is why the cleanup correction is preferable to an operating habit of removing locks whenever another command is blocked. Reliable automatic release reduces the cases requiring an operator to decide whether ownership has ended. It also preserves the refusal to continue when another operation legitimately owns the state.

The release contains other fixes, including provisioner timeout handling and multiple file descriptors that prevented local-state reads on Windows. Those do not change the lock contract. The specific improvement here is that console, import, graph, and push now explicitly release the state lock acquired during backend context initialization and report a failure if that release cannot complete.