Go's checksum database makes dependency versions publicly consistent

A local checksum can detect that today's download differs from yesterday's. It cannot establish that yesterday's download matched what everyone else received. Go's new checksum database addresses that first-download gap by giving module versions public checksum records that clients can verify against a transparent log.

Katie Hockman's August 29 launch announcement declares the module mirror, index and checksum database production ready. The Go command will use the mirror and database by default for Go 1.13 module users. These services solve related problems, but each has a separate responsibility.

The mirror supplies module metadata and source archives. Authenticated checksum records come from the checksum database, while the index supplies a feed of new module versions available through the mirror. Combining their roles into a single claim that a central server makes dependencies safe would miss the actual security design.

I value the explicitness of Go's dependency model. A version requirement names an input, and a checksum constrains the contents accepted for that input. Making those contents consistent across users improves the meaning of a dependency review, because the reviewed version must remain the same version for another downloader.

Module downloads do not need repository history

The launch article explains why fetching through version control can be expensive. Resolving a version may require fetching a repository's history, including for a transitive dependency that is not itself being built. The information needed by the Go command is more specific than the information a repository clone supplies.

A module proxy exposes an API suited to that narrower request. The command can request a list of versions and then the selected version's information, module file and archive. It need not obtain unrelated repository history merely to answer those questions.

A mirror adds storage to this proxy role. It caches module metadata and source so that it can continue serving a version after the original location becomes unavailable. The public service is proxy.golang.org. Earlier Go versions can opt into it with GOPROXY, as the launch article describes.

For example, a POSIX-shell invocation can direct a module download through the mirror.

GOPROXY=https://proxy.golang.org go mod download

That setting selects the source of the download. It should not be confused with enabling the integrated checksum-database verification provided for Go 1.13. The article supplies a separate manual checking tool for users of Go 1.12 and earlier. Selecting the mirror alone does not add that newer client behavior to an older toolchain.

Caching relies on the model that a particular module version has immutable contents. A mirror serving yesterday's archive is useful only if that archive is still the intended identity of the version. The checksum database makes violations of that identity detectable even when the origin server or a proxy supplies the changed contents.

A local go.sum cannot compare different users' first downloads

The existing go.sum file records SHA-256 hashes for dependency source and module files. Once recorded, these values let the command detect a later download with different contents. This protects a project against unexpected changes after it has accepted a version.

The initial acceptance is the remaining problem. A proxy could serve different code to a targeted user when that user first downloads a dependency. If the command simply records the resulting checksum locally, its later consistency checks preserve the targeted copy. They do not discover that other users initially received something else.

The checksum database design, updated April 24, defines correctness for this purpose as receiving the same code everyone else receives for a module version. This is a deliberately limited definition. It makes review of a particular version meaningful without claiming that the version is free of defects or malicious behavior.

The design also avoids requiring individual module authors to manage signing keys for this mechanism. It authenticates the identity of the version's contents through a shared log. That changes what must be trusted about hosting infrastructure without making a statement about the author's intentions.

The log provides proofs in addition to records

The public checksum database runs at sum.golang.org. The launch article describes its transparent log as a Merkle tree backed by Trillian. Clients obtain the relevant checksum lines and verify proofs before adding the lines to their own go.sum files.

An inclusion proof establishes that a particular record belongs to the tree. A consistency proof establishes that a later observed tree extends an earlier one without removing its records. These are different questions. A record's membership alone would not establish that the server has preserved previously accepted history.

The /lookup endpoint returns the requested checksum record and a signed tree head. The /tile endpoint supplies chunks of tree data used for verification. Tiles also allow caching, so many clients can reuse the same pieces of proof material instead of requiring a unique response for every lookup.

The design describes a client retaining its latest observed signed tree size and hash. Clearing cached module data should not erase that remembered state. Otherwise, a routine cache cleanup could also remove the information needed to notice an inconsistent history on a later request.

This design makes the checksum server accountable for what it signs and serves. Third-party auditors can inspect the log, and participants can compare observed states. The launch announcement encourages that independent auditing. It does not claim that a cryptographic data structure eliminates every possible compromise without observation by clients and auditors.

Availability and content identity remain different guarantees

A compromised origin server can withhold a dependency, and a mirror can fail to answer. Checksum verification does not make either server available. A cache can improve availability by retaining an archive, while authentication prevents a changed archive from silently taking its place.

Conversely, successful downloading does not establish authenticity by itself. The database design explains that an unavailable checksum record must not automatically cause the client to accept an unchecked version. Otherwise, an attacker could block verification traffic and turn that failure into a way to substitute code.

Even a module author moving a tag would change the contents associated with the version and trigger the relevant consistency problem. This is intentional. The security mechanism depends on version identity remaining stable after publication, regardless of who attempts to change it.

I find this separation useful for operating build infrastructure. A mirror can be selected for its availability and placement, while the checksum mechanism supplies independent evidence about the downloaded contents. The mirror does not need to become the sole authority on which code belongs to a public version.

Private modules need separate routing and verification decisions

The design discusses privacy because looking up a checksum sends a module path and version to a server. A private path may itself contain sensitive information even when the source is inaccessible. Authentication failures cannot undo the fact that an unintended lookup already transmitted the path.

It therefore treats private-module configuration as part of correctness. Its proposed controls distinguish avoiding a proxy from avoiding the public checksum database. A company might fetch private modules through an internal proxy while excluding them from public checksum lookup. Another user might fetch public code directly while still authenticating it against the database.

These are separate decisions because source routing and content verification solve separate problems. The launch announcement directs users of non-public modules to the environment configuration documentation. Applying public defaults indiscriminately to private names would ignore the privacy concern that the design explicitly identifies.

The proposal also explains that a proxy can relay checksum database requests. That makes it possible to cache verification material and reduce direct contact between individual clients and the public database. Such a proxy changes the network path while clients still verify the signed log evidence.

The index helps other tools find versions

The third launched service, index.golang.org, publishes a feed of new module versions that become available through the mirror. Tools maintaining their own caches can follow this feed instead of discovering every version independently. The index's role is discovery, and its entries do not substitute for checksum verification of an archive.

The services now give module consumers distinct mechanisms for retrieving code, checking version contents and discovering newly available versions. For an engineer reviewing a dependency change, the checksum database's practical contribution is a shared, auditable identity for that version. The review can concern a fixed body of code even when different users download it through different mirrors.