Docker expands BuildKit access and separates build credentials from inputs

A build may need permission to read a private dependency without needing that permission in its final image. Treating a credential as an ordinary source file confuses those lifetimes. Docker 18.09 adds BuildKit session capabilities for build-time secrets and SSH forwarding, while allowing BuildKit builds without experimental daemon mode. The November 8 release notes describe those changes alongside a separate SSH transport for reaching a remote Docker daemon.

The shared terminology can obscure the different mechanisms. Remote daemon SSH transports Docker API traffic to another host. Build-time SSH forwarding lets a build use an agent exposed through its BuildKit session. A secret provider supplies a named file to a frontend that requests it. Each capability addresses a different relationship between the client and the work it asks the builder to perform.

I prefer multi-stage builds whose final image contains the service artifact it needs. Build-time credentials have an even shorter intended lifetime than many intermediate build tools. Giving them a dedicated delivery mechanism makes that distinction expressible in the build interface instead of relying entirely on conventions about ordinary copied files.

BuildKit no longer requires the whole daemon to be experimental

The release removes the experimental daemon requirement for BuildKit. The original July integration required that mode as well as a client-side selection. November's notes also describe a daemon configuration option for BuildKit. The configuration patch in Moby pull request 37593 introduces the associated builder-version information and routing changes.

This changes who can use the backend without opting the daemon into the wider experimental setting. It does not mean that every possible frontend instruction has the same support status as the build API. The frontend parsing a Dockerfile still needs to understand an instruction before the backend can execute its requested operation.

For a BuildKit-capable daemon, an explicit client selection remains easy to recognize. The environment variable applies to this build invocation.

env DOCKER_BUILDKIT=1 docker build -t service:release .

Keeping selection visible is useful when comparing behavior between builders. The Dockerfile identifies the recipe, while the chosen backend determines how its operations are resolved. A build record that identifies both can explain differences in progress output or cache behavior without attributing every difference to a source change.

The release adds docker builder prune for the BuildKit build cache and configurable garbage collection policy. These are operational consequences of retaining intermediate build results. Reusing work consumes storage, so cache management needs an interface of its own. The release notes identify both manual pruning and policy configuration, without implying that application image removal and build-cache removal are the same operation.

The release also names the BuildKit progress modes auto, plain and tty under --progress. Plain output provides an explicit choice for a captured build log, while terminal rendering can be selected separately. This is a presentation choice around reported progress, independent of which secret or SSH resources the session supplies.

Secret delivery uses a named session resource

Tibor Vass's CLI pull request 1288 implements docker build --secret id=mysecret,src=/secret/file. Its description says that the resource is provided to BuildKit frontends requesting the secret under that identifier. This makes the association between a local file and the name used by the build explicit at invocation time.

The patch describes a Dockerfile frontend implementation using RUN --mount=type=secret,id=mysecret. Its original implementation names a particular external frontend. That detail matters historically. Adding the client flag does not establish that an unqualified Dockerfile interpreted by any frontend automatically accepts the mount syntax. The CLI provider and the frontend request must agree.

For the caller, the identifier separates the build's logical request from the credential's location on the client. A build definition can request mysecret, while the invocation chooses the file that satisfies it. This avoids making a workstation-specific credential path part of the source recipe. It also means that supplying a secret is an explicit act of the client running the build.

The secret mechanism does not excuse a build command from handling the supplied content carefully. A process given access to a credential can copy it into an ordinary output or print it. The dedicated input mechanism expresses how the credential arrives, while the command determines what it does with the bytes. This is a consequence of executing a process with access to the resource, not a claim that the release prevents every form of disclosure.

The useful design change is the distinction between the credential resource and the ordinary build context. A dependency-fetching step can request the former without requiring the application source directory to contain it. That gives build authors a clearer description of the permission the step needs and the artifact it is supposed to produce.

SSH forwarding supplies an agent through the build session

The SSH forwarding patch in CLI pull request 1419 adds an SSH agent provider to the BuildKit session. Supplied identifiers and optional paths become agent configurations in the CLI. The release notes describe exposing an agent socket through the --ssh build option.

For a client with an agent socket in SSH_AUTH_SOCK, the invocation can associate it with the default identifier. The Dockerfile frontend must request the corresponding SSH resource for a build step to use it.

env DOCKER_BUILDKIT=1 docker build \
  --ssh "default=$SSH_AUTH_SOCK" -t service:release .

This is a different operation from copying an SSH private key into the build context. The client configures a provider that participates in the session. A build step's use of that provider also remains different from the SSH connection used to control a remote engine. The agent can be relevant to fetching a dependency even when the Docker daemon itself is local.

The patch's option documentation also allows keys or socket paths, so the provider should not be described as accepting only one resource form. This example deliberately shows an existing agent socket. The identifier and path are parsed by the CLI before it attaches the provider to the session.

I find the separation useful because it keeps two kinds of authority distinguishable. Permission to ask Docker to build an image does not, by itself, identify which private source repositories that build should access. The session resource records an additional input to the build request.

Remote daemon SSH proxies the Docker connection

Akihiro Suda's remote connection patch in CLI pull request 1014 adds ssh:// support to the Docker host option. Its documentation requires public-key SSH access to the remote machine and a Docker 18.09 or later binary on that machine. The remote binary supplies docker system dial-stdio, which proxies the daemon socket through standard input and output.

The client can therefore ask for the remote daemon's container list through SSH. The hostname and user below are illustrative.

docker -H ssh://operator@example.com ps

This uses the SSH connection as transport to the daemon. It does not turn the remote shell into the build definition or move the local command's meaning to an unrelated remote executable. The helper provides a connection to the Docker socket so the client can continue using the Docker API.

The remote Docker binary requirement follows from that mechanism. Reachable SSH alone is insufficient if the remote helper command is absent. Conversely, build-time agent forwarding is not a prerequisite for listing containers through a remote daemon. The two options operate at different stages of the interaction.

Runtime packaging also changes the process relationship

The same release notes state that packaged containerd is now managed by systemd, where earlier versions had the Docker engine manage it. They warn that custom docker.service mount settings, such as MountFlags=slave, can break engine interaction with containerd and prevent containers from starting. This is a packaging and service-configuration change alongside the build work.

That warning illustrates why a release cannot be assessed solely through newly available CLI flags. A host's service overrides can encode assumptions about which process starts another process and which mount view they share. Once containerd has its own service management, those assumptions need to match the documented arrangement.

The November release gives build inputs more explicit delivery paths and lets the CLI reach a daemon through an existing SSH connection. Its individual mechanisms remain identifiable. A secret has a named provider, an SSH build resource has an agent configuration, and a remote Docker connection uses a helper on the daemon host. That information is enough to describe which authority a build or remote operation receives without treating every SSH-related option as one capability.