Distribution moves shared registry maintenance into the CNCF

Registry operators had made changes to Docker Distribution that were staying in small forks. That is the concrete maintenance problem behind Docker's February 4 donation announcement. The project has entered the CNCF Sandbox under the name Distribution, with participation sought from organizations already depending on its code. Moving the project gives those operators a shared place to maintain behavior they otherwise carry separately.

The code underlies the registry in Docker Hub and other registries. Docker describes an extensible Go implementation that replaced an earlier Python registry without content addressed storage. The announcement names prospective maintenance participation from Docker, GitHub, GitLab, Digital Ocean, Mirantis and Harbor, with an OCI maintainer also invited. Incubation is a proposed next step, rather than the status announced today.

I care about this because Docker's useful unit of delivery is the image artifact. Describing a repeatable build is only part of making an environment transferable. The system that stores and distributes the resulting bytes needs dependable shared behavior too.

A registry separates names from content

The existing Registry HTTP API V2 specification explains the storage model. A digest identifies content through a collision-resistant hash of its bytes. The identifier includes the algorithm and its hexadecimal result, so a client can calculate the same value independently. The specification recommends SHA-256 for compliant implementations.

That gives an image distributor a useful division of responsibility. A registry tells the client which objects belong to an image, and the client can verify downloaded content against the expected identifiers. Correctness does not depend solely on receiving a successful HTTP response. The bytes need to match the object the client intended to retrieve.

The specification makes an especially important distinction around the Docker-Content-Digest response header. When fetching content by digest, the client must verify it using the digest it requested. A server-supplied header cannot replace that expectation. Otherwise, a server could return different content together with a matching digest for the wrong content, defeating the reason for addressing the request by digest.

This property concerns integrity relative to an expected identifier. It does not by itself establish who produced an image or whether the program inside is appropriate to run. The identifier has to reach the client through a trusted process. The protocol documentation explicitly conditions its argument about retrieving content from an insecure source on communicating that identifier securely.

These details explain why registry behavior is worth sharing across implementations and operators. A small change to digest handling can affect the meaning of verification. Keeping a common implementation maintained allows fixes in that logic to benefit several services, although each service still has to adopt the corrected code.

The manifest describes the artifact

The schema 2 manifest documentation describes an image as a configuration object and an ordered collection of layers. Both the configuration and each layer have descriptors containing a media type, size and digest. Layers start with the base image, so ordering is part of interpreting the image correctly.

The size field gives the client an expectation before validating the downloaded object. According to the documentation, content whose length does not match that expectation should not be trusted. Its digest then identifies the content itself. A registry therefore handles structured references to byte sequences, with explicit information about the representation of each object.

A manifest list adds another level of reference. It points to image manifests for particular platforms, including operating system and architecture. A client distinguishes a manifest list from an image manifest through the returned content type. The list lets a shared image reference describe several platform-specific images without pretending that one executable artifact runs on every machine.

That arrangement makes distribution a separate problem from building. A build produces an image for a particular configuration and platform. The registry needs to preserve the relationships among manifests, configuration objects and layers while serving clients that understand different representations. The manifest documentation identifies multi-architecture support and movement toward content-addressable images as the two principal goals of schema 2.

I prefer small service images because the final artifact should contain what the service needs. The registry model supports that preference without requiring every image to have the same construction. A single-binary image and an image with several filesystem layers still use explicit descriptors to identify the objects being transferred.

Compatibility includes HTTP behavior

Representation negotiation creates obligations for maintainers. Clients advertise support for the schema 2 image manifest and manifest list through the Accept header. They then inspect Content-Type to determine what the registry actually returned. Treating every successful response as the same JSON structure would lose that distinction.

The documented backward compatibility path is more involved than storing two file formats. When a client does not advertise support for the newer representation, the registry can rewrite an image manifest into the older format. For a manifest list, it selects an appropriate Linux amd64 image before doing any needed conversion. The documentation specifies a 404 response if no suitable image exists.

Conversion also exposes a difference in how the formats describe configuration. The old representation expects configuration for each layer, while the newer one provides a single image configuration. The registry creates synthetic configurations to fill the older parent chain. Those synthetic objects do not independently describe runnable images.

This is substantial shared compatibility code. An operator can expose a registry through its own service while inheriting behavior that existing clients rely on. A private fork that changes negotiation has consequences beyond a local storage decision. Contributing those changes upstream gives maintainers an opportunity to examine their effect on other clients before several implementations diverge.

Transfers and storage are separate concerns

The API also addresses interrupted transfers and duplicate uploads. Its resumable push use case lets a client send the remaining portion of a partially uploaded layer after connectivity returns. Resumable pulls use HTTP range requests to avoid downloading bytes already received. These mechanisms matter because the registry distributes objects that can be expensive to send again.

Layer upload deduplication uses knowledge of existing content. When a layer is already present, the registry can tell another uploader that the data need not be sent again. The specification also describes concurrent identical uploads, where both can proceed and the first completed object is stored. Content identity is useful here because the server can recognize the same layer independently of which build produced it.

Storage placement remains configurable. The registry configuration reference describes a YAML configuration file and environment variables for overriding individual values. For example, this is the documented filesystem storage section:

storage:
  filesystem:
    rootdirectory: /var/lib/registry

The corresponding environment variable is REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY. Its underscores represent configuration nesting. The reference recommends using environment variables to adjust individual settings in a base configuration, and discourages overriding whole sections that way.

The same configuration documentation includes object storage backends, including S3, Azure and GCS. This is a practical consequence of the extensible library design. Registry services can share protocol handling while choosing a storage system suited to their infrastructure. Keeping those choices behind a common implementation gives operators a reason to collaborate even when their deployments differ.

Even discovering protocol support has defined behavior. The API specifies a request to /v2/, with a successful response indicating support. An authentication challenge can also indicate support while requiring the client to follow the WWW-Authenticate header and retry. This separates understanding the registry protocol from having permission to access its resources. An operator changing authentication integration therefore still needs to preserve the discovery behavior expected by clients.

The donation changes where that work meets

The announcement identifies a gap between widespread dependence and upstream participation. A shared implementation can be widely deployed while the people adapting it have little involvement in its maintenance. The cost appears when a change remains in a fork and needs continued reconciliation with upstream work. That is the maintenance consequence of the fork pattern Docker describes, rather than a claim that every local modification belongs upstream.

Moving Distribution into the CNCF creates an organizational setting for broader participation, but the useful engineering work remains specific. Maintainers need to preserve digest verification expectations, negotiate manifest representations and support the storage configurations operators use. The existing implementation already contains those responsibilities. The donation gives the organizations that depend on them a shared project in which to maintain their changes.