October 2, 2017·4 min read·engineeringterraformproviderscaching

Terraform adds a shared cache for provider binaries

Terraform 0.10.7 can reuse downloaded provider binaries across working directories. Version selection still consults the distribution server, and cache storage has a separate role from plugin discovery.

Separating provider distribution makes a Terraform working directory responsible for obtaining its plugins. That is convenient for an individual configuration, but the same provider can then be downloaded again for every other configuration on the machine. Terraform 0.10.7 adds a shared cache to reduce that repeated transfer without changing which plugins a configuration requires.

The October 2 changelog introduces optional caching in a shared directory. Its scope is provider binaries. It does not replace state storage, module installation or the configuration's provider version requirements.

I value Terraform's reusable API abstractions, and independent provider releases make those abstractions easier to maintain separately. A cache addresses a practical consequence of that distribution model. Several configurations can use the same binary without each paying the cost of downloading it again.

The working directory remains self-contained

The provider cache documentation shipped with 0.10.7 explains the default behavior. Initialization downloads plugins into a subdirectory of the working directory. Each configuration therefore has its own prepared set of plugins.

The cache adds a location from which those local installations can obtain an already downloaded binary. If initialization chooses a version absent from the cache, Terraform downloads it there first and then installs it into the working directory. A subsequent configuration can reuse that cached download.

This keeps the distinction between shared storage and a configuration's prepared dependencies. The cache can contain versions that a particular configuration does not use. The directory's contents do not describe that configuration's intended provider policy.

That separation also helps explain why deleting an unused cache entry is a storage decision. It does not edit a provider constraint in the infrastructure configuration. A later initialization that selects the missing release can download it again.

Configure caching at the CLI level

The cache directory belongs in the CLI configuration file through plugin_cache_dir. The documentation explicitly distinguishes that file from the .tf files used to describe infrastructure. A cache path is a setting for the local Terraform installation.

plugin_cache_dir = "$HOME/.terraform.d/plugin-cache"

This is the documented form for a persistent cache setting. It does not belong inside an AWS or other provider block. A provider block describes the provider instance and its requirements, while the CLI setting controls where downloaded binaries can be reused.

Terraform also accepts the TF_PLUGIN_CACHE_DIR environment variable. The reference describes it as a way to enable the cache or override a configured path for a shell session. That is useful when an automation environment supplies its own storage location without editing each infrastructure repository.

On Windows, the documentation requires forward slashes in the configuration path because a backslash begins an escape sequence in the parser. This is a configuration-language detail, separate from whether the filesystem normally displays paths using backslashes.

A cache hit does not eliminate version discovery

The most useful qualification is that initialization still contacts the plugin distribution server for metadata. Terraform selects an appropriate version and then checks whether the corresponding binary already exists in the cache. A hit avoids downloading the binary again.

The cache consequently does not turn ordinary initialization into a fully offline operation. Having some provider binaries locally is not the same as having all the information that this initialization path uses to choose one. The documented behavior retains the metadata request.

It also means the cache is not a version pin. A configuration with a permissive provider constraint can select a newer release that is not yet cached. Terraform can download that release and add it alongside the existing versions. The cached older binary does not force selection to stay with it.

I prefer keeping those responsibilities separate. A constraint states acceptable code versions, while caching reduces transfer for a version already selected. Using a cache as an accidental dependency policy would make a configuration depend on unrelated details of a machine's download history.

Binary reuse can also reduce duplicate storage

When possible, Terraform uses hardlinks or symlinks so different working directories do not each need a separate stored copy of the cached binary. The 0.10.7 documentation says this optimization is not supported on Windows, where Terraform always creates a copy instead.

That qualification distinguishes avoiding a network download from avoiding duplicate disk storage. A cache can provide the first benefit even when the platform requires a separate local file. Claims about disk savings therefore need to account for how Terraform installs the cached binary on that platform.

The cache must also be separate from directories used for preinstalled plugin discovery, including the third-party plugin directory. The reference says cache management conflicts with normal discovery when both operate in the same directory. A collection of manually supplied plugins and a cache may contain similar filenames, but they serve different initialization paths.

Cache retention is explicit

Terraform does not delete cached plugins automatically. Over time, upgrades can leave several versions that no current configuration uses. The documentation assigns cleanup of those entries to the user.

That policy makes reuse predictable across configurations whose upgrade schedules differ. Removing an old version automatically after one configuration upgrades could make another configuration download it again. Retaining versions avoids that particular coupling, at the cost of requiring storage cleanup.

The cache therefore adds a specific optimization to independent provider installation. Configuration still determines acceptable providers, initialization still chooses versions, and each working directory still receives its required binaries. The shared directory lets those installations reuse downloads while retaining multiple releases for configurations that need different versions.