Docker 18.09.6 fixes stale cache reuse for overlapping filenames

Gemfile and Gemfile.lock are separate files, although one name starts with the other. A BuildKit optimization failed to preserve that distinction when deciding which source paths needed checksumming. Docker 18.09.6, released May 6, includes the correction for multi-source COPY and ADD operations. The failure could preserve a cached result after an input changed, which is a correctness problem for the resulting image.

The Engine release notes link Moby issue 38964. That issue describes a changed source file failing to invalidate the BuildKit cache. Its discussion traces the practical case to a dependency lockfile and identifies a smaller BuildKit patch as the correction. Reading the issue is necessary here because a short phrase about cache invalidation can otherwise be read as a performance fix in the opposite direction.

I consider Dockerfile cache behavior part of the build's semantics. Ordering instructions to retain useful cached work is valuable only if a changed input also discards the work that depended on it. A fast rebuild with old dependency contents fails the purpose of a repeatable build.

A string prefix is not a parent directory

The BuildKit correction in pull request 858 changes selector deduplication. Its description says the issue appears when one checksummed path contains another. The diff makes the intended filesystem interpretation precise by requiring a slash after the candidate parent path.

Before the correction, the deduplication code could discard one path when its string began with another path's string. That is appropriate for a child path only if the prefix ends at a directory boundary. Gemfile.lock begins with Gemfile, but it does not reside inside a directory named Gemfile. Dropping it from the selectors loses a distinct input.

The new test suite preserves both names in either input order. It separately tests that Gemfile/foo can be removed when Gemfile already covers it. Another case preserves sibling paths under the same parent when that parent itself is absent. These examples define the optimization's intended behavior without depending on a particular dependency manager.

The rule is small enough to state directly. A path can subsume another selected path when it is an ancestor directory. A shared textual prefix alone does not establish ancestry. The patch retains deduplication while correcting the boundary used to determine which inputs it may remove.

Why the lockfile made the failure visible

The Moby report initially supplies a synthetic reproduction with two input files. A maintainer cannot reproduce the unexpected result in that simplified case. The conversation then returns to the actual dependency inputs, where an update to Gemfile.lock leaves a build step cached and the application build cannot find the expected gem version.

This sequence matters because the simplified names can remove the bug's prerequisite. Two unrelated filenames exercise a multi-source copy, but they do not necessarily exercise the overlapping-prefix behavior. The issue becomes understandable when the relationship between the original filenames is retained.

The March 28 discussion identifies the BuildKit correction and supplies an immediate workaround. Put the two files in separate instructions so the affected multi-source selector combination is avoided.

COPY Gemfile .
COPY Gemfile.lock .

Those instructions come from the issue discussion. They illustrate the workaround, rather than a complete application Dockerfile. Their purpose is to separate the file-selection operations without discarding the build cache globally. The maintenance release supplies the underlying correction so a multi-source instruction can again account for both inputs.

Cache correctness differs from cache efficiency

An unnecessary cache miss repeats work that could have been reused. A false cache hit reuses work whose inputs have changed. Both are cache bugs, but they have different consequences and demand different descriptions. The linked report concerns the latter behavior.

The path-selector change explains why. Removing a distinct source from the checksummed selection can make a relevant content change invisible to the cache decision. Retaining that source allows the changed content to participate again. The correction therefore makes invalidation happen when required, rather than broadly preventing invalidation.

That is especially relevant for dependency inputs copied before application source. Such a Dockerfile intentionally separates relatively stable dependency work from more frequent source edits. If a lockfile change does not affect the dependency step, the build can combine a new declared dependency set with old installed contents. The optimization has then broken the relationship the Dockerfile was designed to express.

My preference for a minimal final image does not make that mismatch easier to detect. A compact runtime image can faithfully copy the output of an incorrectly cached build stage. Reducing the files shipped to production and correctly identifying the inputs of the build stage are independent requirements.

The useful scope of this maintenance release is therefore quite concrete. It restores correct handling of separate source files whose names share a prefix, while preserving deduplication for genuine descendants. The tests capture that distinction with both filename orderings and with parent-directory examples. A dependency lockfile once again contributes to the cache decision as a separate file.