Nix restores parallel copying while respecting store dependencies
Nix 2.11 restores concurrent copying for stores that use the general import path. Dependency-aware scheduling and existing batch protocols explain where the change applies.
Copying a Nix closure can involve many store paths whose transfers do not all depend on each other. Serializing every transfer adds waiting that the dependency graph does not require. Nix 2.11 restores parallel copying where the destination supports the general per-path import mechanism. The August 24 release notes explicitly exclude the daemon and ssh-ng stores, which already copy everything in a batch to avoid latency problems.
This is a restoration of concurrent execution, not a new closure format. The interesting part is how a graph of store references constrains copying. Independent paths can progress together, but the implementation still needs an order compatible with the destination's dependency requirements.
Copying operates on closures
The 2.11 command manual defines nix copy as copying store path closures between two stores. --from selects the source, and --to selects the destination. An omitted side defaults to the local store.
A local binary cache provides a small example of the destination syntax.
nix copy --to file:///tmp/example-cache nixpkgs#hello
The file:// scheme matters. The manual distinguishes a file-backed binary cache from a bare filesystem path, which selects a chroot store instead. Two arguments that look like locations can therefore select different store implementations. Understanding that selection is necessary before deciding which copying behavior applies.
The example assumes the experimental command and flake interfaces are enabled. It does not claim that the package is only one file or that the destination receives only its executable. The command's closure semantics account for the store paths required by the selected result.
I value this because a declarative package definition is useful beyond the machine that built it. Moving the complete result to another store is part of making that definition operationally useful. Transfer scheduling should follow the actual dependency requirements without adding unnecessary serial work.
The implementation finds missing paths first
The store-copy implementation in 2.11 queries the destination for valid paths and forms the set that remains missing. It then topologically sorts the missing source paths before constructing the sources used for import. The source comments explain that the general multiple-path interface expects a sorted list.
This avoids treating already available paths as transfers that must run again. It also makes the dependency information explicit before the destination starts consuming the import sources. A repeated copy can therefore have a different amount of transfer work even when the requested top-level result is unchanged.
The general addMultipleToStore implementation then uses a thread pool and graph processing. Its dependency callback returns the references of each path. That supplies the scheduler with the relationships it must respect while allowing independent work to proceed.
The implementation checks validity again before importing a path. That second check is useful in a concurrent system because the destination's state can change between the earlier missing-path query and the eventual import. An initial absence does not need to force another import after the path has become available.
Parallelism follows paths with usable dependencies
Consider a closure in which two independent libraries are both needed by an application. The libraries provide potential concurrency before the application can be imported. A long chain of references offers less concurrency because each path's dependencies constrain its readiness. The same number of paths can therefore imply different scheduling opportunities.
This is why the release's qualification about copying in parallel as much as possible matters. It does not promise a fixed acceleration or that every path transfers simultaneously. The reference graph, destination behavior, and available transfer resources all limit the work that can overlap.
The source tracks completed, running, and failed work for progress reporting. With keep-going disabled, an import error propagates. With it enabled, the implementation records and reports a copy failure while following the continued-work policy. Concurrent progress does not turn an unsuccessful path import into a successful closure transfer.
For an operator, that means a progress count and a completed command answer different questions. Several paths can finish while another fails. The command's error behavior remains part of deciding whether the intended result is available at the destination.
Existing batch stores keep their own strategy
The release notes say the restored general parallel copying does not apply to daemon and ssh-ng. Those implementations batch the operation to avoid latency issues. Splitting a batch into many separate calls can incur communication costs even when the resulting calls are concurrent.
The command manual also documents destination substitution for SSH stores through -s. That allows the destination to obtain missing paths from its substituters. It can change which network route carries the data, independently of the general per-path scheduling change.
My preference is to explain a transfer in terms of its source, destination implementation, and dependency graph. A claim that all Nix copies now become faster would omit those distinctions. The August release restores a specific opportunity for independent paths to transfer concurrently while retaining batch behavior where the store protocol already handles the operation as one request.