NixOS 17.03 composes package changes with overlays

A package customization needs two views of the package collection. It needs the recipe being changed, and it needs the dependency choices that should apply after all customizations have been considered. NixOS 17.03 introduces overlays that name those views separately. The release notes shipped with Gorilla, dated March 31, 2017, announce the mechanism and replace the former overridePackages interface. The distinction is useful for local modifications that need to remain consistent with one another.

The release is based on Glibc 2.25, GCC 5.4.0 and systemd 232, with Linux 4.9 as its default kernel and Nix 1.11.8 as its package manager. Those version changes describe the common base. Overlays describe how a user can extend the package collection without treating every local difference as an unrelated copy of a package definition.

I am drawn to declarative systems because they make intended state available for inspection. Overlays fit that preference when their dependency choices are explicit. They also require care about which package set an expression refers to, so understanding the two arguments matters more than memorizing an import example.

Previous recipes and final dependencies

The overlay documentation in the release describes overlays as layers in the fixed point used to compose Nixpkgs. Each layer is a function of two arguments returning a package set. The conventional names are self and super.

self is the final package set. The documentation says to use it for dependencies of packages defined in the overlay. super is the result of evaluating earlier stages. It excludes packages added by the current layer and later layers, making it the place to obtain an existing recipe that is being overridden.

The documented Boost example reduces to:

self: super: {
  boost = super.boost.override {
    python = self.python3;
  };
}

The starting Boost recipe comes from the previous stage. Its selected Python dependency comes from the final package set. This lets the customization participate in other package choices instead of hard-coding a dependency from a separate, partially customized collection.

The names are conventions, but the distinction is part of the interface. Replacing super.boost with the final self.boost would ask for the very result the overlay is defining. Looking up every dependency through super would instead restrict those choices to earlier stages. The two views serve different purposes within one definition.

Layer order is an input

An import can receive an explicit overlays list. The release also supports loading a directory found through the nixpkgs-overlays search-path entry or the user's ~/.nixpkgs/overlays/ directory. The first present source wins, and later sources in that search order are ignored.

Directory loading has its own ordering rule. Files, directories containing default.nix and symlinks to those forms can supply overlays. Entries load alphabetically. The documentation explicitly warns that layer order can influence package recipes when multiple layers override the same recipe.

That makes overlay discovery part of the environment needed to explain a package set. An explicit list states the chosen order directly. An automatically discovered directory states it through its contents and names. Neither mechanism implies that independently written layers commute when they modify the same package.

The replacement for overridePackages uses the collection's path and supplies overlays during import. Existing code using the removed function must change its construction of the package set. This is an interface migration, not simply a new optional spelling that leaves every previous expression valid.

I prefer an explicit list when a configuration's package choices need to be reviewed together. That keeps ordering close to the import that consumes it. Directory discovery is convenient, but its precedence and alphabetical ordering are still inputs to the result and should be understood as such.

Cross compilation also changes package expressions

The release rewrites cross compilation. Derivations no longer expose .nativeDrv and .crossDrv, and the notes describe them as cross by default instead of native in that context. Existing expressions that select one of those attributes therefore need to follow the new interface.

This matters alongside overlays because both changes concern how package expressions refer to other derivations. A local overlay is not exempt from changes to the package collection it extends. Keeping a customization separate from upstream files reduces editing overlap, but it does not freeze the semantics of every interface used in that customization.

The stdenv.overrides function also changes its argument contract to self and super. That brings another customization point into the two-view model. The release notes direct readers to lib.trivial.extends for the meaning of those parameters, tying the convention to the mechanism used to extend package definitions.

These changes make 17.03 more than a set of newer package versions for people maintaining local expressions. The shape of the customization interface changes as well. The source of a recipe and the dependency set into which it is composed now have explicit roles in the documented overlay model.

System identity retains allocation history

NixOS also changes user and group handling to retain deallocated UID and GID information. When a user or group is revived, that record allows the previous identifier to be allocated again. The notes state that identifiers are consequently no longer reused.

This is an important qualification to a purely stateless account of declarative administration. A current configuration says which accounts should exist. Allocation history supplies information about identities that existed before. The new behavior preserves that information so a temporarily absent account does not automatically lose the association with its old identifier.

The practical purpose is continuity of identity across configuration changes. Deleting a declaration and later restoring it can now refer to the previously allocated identifier. That is a different operation from making the identifier available for an unrelated account. The release explicitly distinguishes the two through its retained allocation state.

Defaults can change without changing a local declaration

The default time-synchronization service changes from Ntpd to systemd-timesyncd. Setting services.ntp.enable = true restores the earlier service choice. Upstream time servers are now configured through networking.timeServers for all NTP implementations, which separates server selection from the particular implementation providing synchronization.

Firewall defaults change as well. Automatic loading of connection-tracking helpers is disabled. The notes describe the previous behavior as insecure if the firewall is not configured properly and identify active FTP as an example that may need helpers. The relevant options allow enabling the behavior and selecting connection-tracking modules deliberately.

These are consequential changes for an otherwise unchanged configuration. A declaration that relies on defaults inherits their new values from the release. Declarative management makes that dependency inspectable, but it does not make defaults permanently stable. Explicit overrides remain necessary where the local system requires a particular choice.

The release similarly stops giving Postfix an empty local_recipient_maps value by default. PHP scans extra configuration in /etc/php.d instead of the broader /etc directory, preventing unrelated .ini files from being loaded. These changes narrow behavior that had previously followed a more permissive default.

The module type system also gains extensible option types. Several modules can extend a type such as an enumeration through declarations of the same option. That supports composition at the configuration interface as well as at the package level. An enumeration can acquire its valid choices through the modules participating in the configuration, instead of requiring every choice to be collected in one original declaration.

Package policy can retain a reason

Nixpkgs can now mark packages insecure through listed vulnerabilities. That adds information about why a package has a restricted status, beyond the existence of a particular version in the collection. The release separately removes Ansible 1 because upstream has not patched a serious vulnerability, making Ansible 2 the default.

This is another point where a package definition carries policy as well as build instructions. A configuration selecting a package needs to account for that policy, while an overlay changing its recipe still participates in the same package collection. Local customization can express a deliberate choice, but its existence alone does not explain the security status of the selected software.

The release also splits the BIND DNS utilities into their own output and exposes them through pkgs.dnsutils. A system needing dig no longer has to pull in the whole BIND package for that purpose. This is a concrete reduction in the dependency needed for an administrative tool, and it remains useful independently of whether the system uses the new overlay interface.