NixOS measures reproducibility across the minimal ISO build

A reproducibility percentage needs a definition of what was rebuilt. Graham Christensen's new NixOS tracker announcement reports 98.473 percent for the unstable minimal x86_64 installation image. In a February 6 clarification, he explains that the work rebuilds every derivation involved in producing the ISO. The measured population includes build tools, not only files ultimately placed in the image.

That distinction makes the result more useful. A compiler or test dependency can participate in constructing an image without belonging to its installed contents. Rebuilding the derivations involved in production examines a wider construction process than a comparison limited to the final filesystem. The percentage must be read against that population.

The declarative model is what attracts me to Nix, and this sort of measurement strengthens it. Recording inputs provides a basis for repeating a build. Actually performing the repetition supplies evidence about whether those inputs determine the output under the conditions exercised.

The scripts identify the measured job

The tracker's period check script instantiates the nixos.iso_minimal.x86_64-linux job in release-combined.nix. It then queries that derivation's requisites and selects paths ending in .drv. Those paths define the build jobs it iterates over.

This explains the appearance of packages such as pytest and ninja that prompted a question in the discussion. Their presence in a build graph need not mean they were shipped as installed tools on the ISO. The relevant question is whether their derivations participate in producing the selected job.

The script also records the Nixpkgs revision in the log filename. That connects a result to a particular expression set. A report for one revision is evidence about that revision's graph, not an unqualified percentage for every package that happens to be available through Nixpkgs.

The announcement says the tracker will rebuild automatically each week. Repeated reporting matters because the set of inputs and package expressions changes. A single successful comparison can establish an observation for one run, while a series of recorded runs can expose a regression after the graph changes.

A check rebuild compares with an existing output

For each derivation, the script first performs a normal build. It then invokes a check rebuild with --check and --keep-failed. The report generator at the same revision includes a small package example of that workflow.

nix-build . -A hello
nix-build . -A hello --check --keep-failed

This example assumes the current directory contains the package expression providing hello. It shows the mechanism used to compare a rebuilt result, not a command that reruns the whole ISO tracker. When a check differs, the report describes a separate .check path that can be inspected against the existing output.

The checker handles outcomes separately. A failure during the first build receives a failed-on-first-build status. Successful checks receive a reproducible status. A failing check takes the unreproducible reporting path and attempts to generate differences when a retained check output exists.

That distinction matters when interpreting the report. A build that did not complete cannot demonstrate matching output. The script's status names and control flow provide the basis for understanding what contributes to the result, rather than treating a percentage as self-explanatory.

Differences become evidence for a package fix

When a check fails and a retained output is available, the script runs diffoscope and writes an HTML difference report. The report generator links those comparisons from its list of unreproduced paths. This connects the headline result to particular artifacts that a maintainer can examine.

A useful failure report needs more than the statement that two outputs differ. A detailed comparison can show where the difference occurs and guide investigation of the package's build procedure. The tracker supplies that next piece of evidence instead of leaving the percentage disconnected from actionable outputs.

Christensen says in the announcement that he has already sent patches for Perl. That is a statement about work submitted at this point, not a basis for claiming every Perl-related result is now reproducible. The tracker records observed results, while a package patch has its own review and inclusion process.

I find that connection between measurement and a specific patch more persuasive than a general claim that declarative builds must reproduce. The package manager can arrange the repeated computation. A package's own build still has to avoid allowing uncontrolled differences to affect its artifacts.

A matching result has defined limits

The period report says builds run at different times on different hardware with different kernels. It also states that the test does not currently inject randomness through filesystem ordering. The report proposes disorderfs as a possible next step to vary directory entries.

Those details define the exercised conditions. Matching across the observed environments is useful evidence, but it does not establish that every untested variation leaves the output unchanged. A latent dependency on directory enumeration order can remain invisible if the environments happen to expose the same relevant ordering.

The report's qualification is therefore a property of the experiment, not a reason to dismiss the result. It states which variation has been attempted and which further variation could expose more nondeterminism. That makes the 98.473 percent figure interpretable as an observed outcome.

The denominator prevents a misleading comparison

The report generator computes the percentage from reproducible log entries divided by all entries in the selected log. The displayed number is therefore a count of reported derivation outcomes. It is not a percentage of image bytes, a fraction of all Nixpkgs packages or a probability that any arbitrary NixOS machine can be reproduced.

Comparing it with another distribution's package-wide percentage would require reconciling those populations and procedures. The ISO production graph is a specific subset with its own build dependencies. A project-wide repository statistic answers a broader sampling question even if both reports use the word reproducible.

For this tracker, the useful result is a named build job with revision-specific logs and differences for failures. The ISO gives the work a bounded target, while rebuilding its production derivations identifies packages that can be fixed individually. That provides a concrete route from an aggregate percentage to the artifact whose build needs investigation.