Docker integrates BuildKit as an experimental build backend

A Dockerfile describes a build in a readable order, but the order of its text is not a complete description of the dependencies between its results. Independent work need not wait for unrelated work, and repeated operations need a cache representation that identifies what they actually depend on. BuildKit addresses that distinction by separating the build definition from the machinery that executes it. Docker 18.06 now offers an experimental integration through docker build.

The July 18 release notes specify two requirements. The daemon must run in experimental mode, and the client must set DOCKER_BUILDKIT=1. This is an explicit choice of backend. Installing the release alone does not mean every Dockerfile build starts using BuildKit.

I regard the repeatable build file and layered image format as Docker's central contribution. BuildKit interests me because it keeps a familiar description while changing the machinery that interprets dependencies and cached results. That is particularly relevant to multi-stage builds, where the relationship between outputs is more informative than their position in the file.

The proposal separated language from execution

Tõnis Tiigi's BuildKit proposal in Moby issue 32925, opened in April 2017, describes a separate project for reusable build technology. Its frontend interprets a user-facing definition. Its backend solves a common low-level description of the operations. Dockerfile is one frontend, so replacing the backend does not inherently require a replacement language.

That separation addresses a limitation in treating every improvement as a change to the Dockerfile interpreter. Scheduling a graph, retaining intermediate results and exporting an artifact are different responsibilities from parsing a build instruction. The proposal assigns them to separate components and describes how those components can cooperate during a build.

The design uses a directed acyclic graph of low-level operations. Starting with the requested result, the solver follows its dependencies and resolves the operations needed to produce it. A node's dependencies express the work that must finish first. Independent dependency resolution can proceed concurrently where the graph permits it.

The proposal is a design document, so its descriptions of intended behavior should be read as design goals. The BuildKit README at the revision introduced by the integration provides a contemporary implementation account. It describes concurrent dependency resolution and instruction caching, and explains the low-level representation used by the implemented frontends.

LLB records operations and dependencies

BuildKit calls its binary intermediate representation LLB. The contemporary README describes it as Protocol Buffer messages defining the dependency graph for processes involved in a build. A frontend translates a higher-level description into that representation. The solver and workers can then operate without requiring every execution decision to be embedded in the syntax of the original file.

This is useful even when Dockerfile remains the only language an application author writes. The parser can interpret stages and references between them. The execution system receives the resulting graph and can reason about which input an operation needs. A build file's readability and the backend's execution model can develop separately.

The README's standalone examples make this intermediate step observable. A Go program can generate an LLB definition without executing its build operations, and buildctl debug dump-llb can inspect that definition. Execution is a separate request. This gives tooling authors a concrete way to study what a frontend has produced before asking a worker to perform the described work. It also distinguishes generating a build plan from producing the final artifact.

The original proposal describes operations for executing a process against snapshots, copying data between snapshots, and obtaining source data. Snapshots represent filesystem state as the builder performs operations. A worker carries out the operation, while the solver decides how to satisfy its dependencies. This separation identifies where execution policy belongs without making it part of each source-language instruction.

One practical consequence is that the graph contains more structure than a transcript of shell commands. A shell command still has its own behavior, but the builder can identify the snapshots and dependencies supplied to it. The proposal's cache design uses those operation relationships to associate results with cache keys and reuse a snapshot when the relevant result is already available.

Caching becomes a property of the build graph

The proposal describes computing a cache key while solving an instruction. If the cache already contains the corresponding result, its snapshot can satisfy the operation. After executing new work, the builder records the resulting snapshot under its cache key. The intended benefit is reducing duplicated work across invocations and between steps that produce reusable results.

This explains why the architecture matters beyond a faster progress display. Reusing a result requires identifying what that result depends on. When the build definition is a graph, dependency relationships can participate in that identification. A cache entry can represent a particular operation result instead of being understood only through the preceding textual history of a build file.

The contemporary README also lists cache import and export among BuildKit's capabilities. Its standalone tooling has explicit exporters for different result forms. Those project capabilities should not all be treated as Docker CLI options in this release. The integration has its own API and command path, and its exposed behavior is a subset of what a separate BuildKit client can express.

For Dockerfile authors, the immediate implication is still concrete. A multi-stage recipe describes dependencies that a graph-based builder can use. The choice of backend affects how the recipe's operations are resolved and how their results are retained. It does not establish that an arbitrary command hidden inside a RUN instruction becomes internally parallel or automatically reproducible.

The integration connects sessions and image export

The Moby integration patch 37151 introduces the new builder path and vendors BuildKit. The corresponding CLI integration patch 1111 connects the command to that backend. Both sides need changes because the client supplies local inputs and receives progress while the daemon coordinates execution and records the image result.

The CLI work includes a session for synchronizing local filesystem inputs and providing registry authentication. The daemon integration translates Docker build options into frontend attributes and invokes the Dockerfile frontend. That mechanism preserves the familiar command's role while passing the definition into BuildKit's control and solver machinery.

For a daemon already configured for experimental operation, the client invocation is small. env makes the environment setting explicit for this one command.

env DOCKER_BUILDKIT=1 docker build -t service:buildkit .

The two enablement requirements remain separate. The environment setting selects the client path, while the daemon must support and permit the experimental backend. A client-side setting cannot create a missing server capability. This is an integration across a client-server boundary, even when both processes run on the same development machine.

The resulting artifact remains useful through Docker's image workflow. The integration includes an image exporter and access to Docker's image and layer services. BuildKit's internal representation therefore does not require application deployment to consume an LLB graph. The graph is a way to construct the artifact, and the exporter supplies the image that the surrounding workflow expects.

Project architecture and shipped interface have different scope

The standalone BuildKit README describes OCI and containerd workers, several output formats, and a separate buildctl command. Those are evidence of the project's broader design. They do not mean Docker 18.06 exposes every standalone control through docker build, or that enabling the integration changes the daemon's general container runtime architecture.

This distinction also applies to rootless execution, which the project README lists as a capability with separate documentation. It would be wrong to describe the experimental Docker build toggle as making the Docker daemon rootless. A property of one BuildKit deployment mode does not automatically transfer to the engine embedding it.

I prefer to judge this release by the integration it actually provides. A Dockerfile can enter the graph-based backend through the existing build command, and the resulting image can return to Docker's image workflow. The new architecture makes dependency resolution and cache management independently understandable parts of building that image.

The experimental status gives this release a specific audience. Build tooling authors and engineers with multi-stage recipes can examine how their existing definitions behave under a different solver. The useful comparison concerns which operations execute, which results the cache supplies, and how the produced image relates to the recipe. These questions follow directly from the responsibilities BuildKit separates, so they can be investigated without assuming a universal build-time improvement.