June 25, 2020·4 min read·engineeringterraformbuildsreliability

Terraform restores the maintenance release's intended Go toolchain

Terraform 0.12.28 corrects the Go version used to build the maintenance binary. The reported failure occurred before ordinary configuration work, even during terraform version.

A Terraform failure that also affects terraform version points outside the usual configuration investigation. That is the useful detail in the report behind Terraform 0.12.28. The reporter could trigger a fatal Go runtime error while asking the executable to identify itself, without supplying an infrastructure configuration.

The June 25 release corrects the build toolchain for the 0.12 maintenance line. Its changelog says the binary should use Go 1.12.13 instead of Go 1.14.2, the version intended for Terraform 0.13. This is a correction to production of the executable, rather than a configuration-language addition.

I regard release construction as part of an infrastructure tool's compatibility contract. A reusable DSL is useful only if its executable continues to start on the systems that run it. This incident gives a concrete reason to include the build toolchain when identifying what changed between maintenance releases.

The report narrowed the failure early

Issue 25386 was opened on June 25 against Terraform 0.12.27. The reporter identified macOS Mojave 10.14.6 and supplied stack traces from several invocations. The failure included a runtime network-polling diagnostic and a fatal runtime error.

The report explicitly noted that the crash affected the version command. Its configuration section was inapplicable, and its reproduction steps also named initialization and planning. This separates the observed problem from a particular resource argument or provider schema. A configuration change cannot explain away a crash demonstrated without that configuration.

The supplied traces entered Go runtime scheduling and network-polling code. Other frames showed package initialization. Those observations support a narrow conclusion about where the process failed. They do not establish that every crash in those functions shares this cause, or that every supported platform was affected.

The same issue contains the maintainer's explanation, posted that day. The maintenance release had been built using the Go version intended for the 0.13 line, and 0.12.28 was released to correct it. That explanation connects the observed regression to the release-build mistake without requiring speculation about an undocumented application-level fix.

Source version and build environment are separate inputs

The tagged .go-version file contains 1.12.13. Alongside the changelog, it records the expected toolchain for this release. A source tag identifies the program's source, while the toolchain determines how that source becomes an executable.

That distinction matters when evaluating a small maintenance release. An unchanged configuration language does not imply an identical execution environment. The compiler and runtime selected during a build can affect the resulting program even when no resource-planning behavior was deliberately changed.

The release note names the two Go versions precisely. Replacing that detail with a general recommendation to use the newest compiler would lose the point. The correction restores the build input expected for the maintenance line. The source does not claim that Go 1.14.2 is unsuitable for every program or that all Terraform branches require one shared toolchain.

For a distributor rebuilding Terraform, that makes the release recipe relevant evidence. Matching the Terraform source version while choosing a different compiler is a different build. It may be a deliberate packaging decision, but it should not be confused with reproducing the toolchain choice documented by this maintenance release.

The previous patch addressed a different panic

The same changelog dates 0.12.27 to June 24 and describes a remote-backend connection-error panic fix. The patch for that fix guards access to an HTTP response in a retry logging hook. A connection error can leave that response absent.

Previously, the hook accessed the response status code when deciding whether an error was rate limiting. The patch checks that the response exists before inspecting its status. This is application error handling in the remote backend, separate from the runtime failure reported after the binary was released.

Keeping those causes separate prevents an incorrect reading of the sequence. A release can repair a real application bug and still introduce a build regression. The correction in 0.12.28 concerns the latter. It does not make the earlier connection-error fix unnecessary.

What the release establishes

The documented correction is specific enough to guide incident classification. A crash in the affected executable during a basic startup command belongs with the build regression report. A remote connection problem that reaches the backend's retry hook belongs with the earlier nil-response repair. Similar words such as crash and panic do not make those mechanisms interchangeable.

I prefer release explanations that retain this level of specificity. Naming the expected compiler, the mistakenly selected compiler and a configuration-independent reproduction gives readers evidence they can compare with their own failure. It avoids turning a maintenance update into an unsupported claim about every possible execution problem.

Terraform 0.12.28 consequently has a small changelog with a substantial operational purpose. It restores the intended Go toolchain for the maintenance executable while retaining the preceding release's application fixes. The useful compatibility information is in how the binary was built, even though the Terraform configuration accepted by the release is not the subject of the change.