Swift tests compiler changes against community projects

A compiler's own tests cannot contain every program its users depend on. Swift's new source compatibility suite adds community projects to the evidence available when the compiler changes. Its central mechanism is straightforward. Keep a known revision of a real project, establish which released language version accepts it, and try building it with the development compiler.

The April 24 announcement describes periodic builds in Swift's continuous integration system. Compiler developers can also request the suite against a pull request before merging. That places compatibility checking at two points in development. Periodic runs can detect a regression already present in development, while requested runs can expose a problem before the responsible change enters the shared branch.

I find Swift's language design appealing, but compiler upgrades impose a cost when they interrupt application work. A collection of real dependencies is useful evidence for a compatibility promise. It also makes the promise more specific, because a failing project provides source that another contributor can inspect.

A pinned project provides a comparison

The suite's launch-day documentation describes a JSON project index. Each entry identifies a public Git repository, a local path and a maintainer. Its compatibility section associates a Swift version with a commit known to compile under that version. The index can record multiple commits for different language versions.

Pinning matters because a moving application branch would change the experiment. A failure after a compiler update could otherwise come from a simultaneous change in the application. Recording the commit and its known compatibility gives the comparison a defined input. The suite can ask whether the compiler still accepts that source without assuming that today's application branch is identical to yesterday's.

The index also describes actions. A Swift Package Manager project can request a release build and a package test action. Xcode entries can name a workspace scheme or a project target, along with configuration and destination. Compatibility testing therefore includes instructions for invoking the project instead of assuming that every repository has the same build layout.

The documentation permits removing the test action when tests are unsupported. This means a successful result must be interpreted according to the actions actually requested. A successful build establishes that the configured build succeeded. A test action provides additional evidence from the project's tests. Neither result establishes behavior for configurations or destinations that the entry does not run.

Inclusion has practical constraints

The initial acceptance rules require a public repository and a project that builds through Xcode or Swift Package Manager. Carthage and CocoaPods are not supported at launch, although the documentation says support is being explored. The project must support building on Linux or macOS and target Linux, macOS, or the listed Apple device platforms.

Those rules constrain the initial evidence. An application whose dependency installation requires an unsupported integration does not become covered merely because its source is Swift. The suite also requires compatibility with the latest GM or beta versions of Xcode and SwiftPM. Maintaining a reproducible entry therefore includes keeping its build instructions usable with the tools running the suite.

Projects need a branch that builds in Swift 3.0 compatibility mode and passes any unit tests. The submission instructions ask maintainers to choose a commit that builds against Swift 3.0 GM, then verify the indexed actions against the specified version. This establishes the baseline before the development compiler is blamed for a failure.

Linux has a particularly important qualification. The documentation accepts Linux projects but says Linux compatibility testing in continuous integration is not yet available. Acceptance into the index and an automated platform run are separate conditions. Reading the launch as complete automated Linux coverage would overstate what the project has announced.

Maintainers share responsibility for useful failures

The suite is community driven, and inclusion requires maintainers who commit to resolving issues promptly. The documentation gives them two weeks after notification to update a project when a Swift change legitimately breaks compatibility, such as a compiler correction to previously wrong behavior. Unmaintained projects may leave the index.

That policy acknowledges that a compatibility failure needs interpretation. A compiler can regress and reject valid source. It can also correct behavior that should never have been accepted. The project entry supplies a reproducible program, but the maintainers and compiler contributors still have to decide which explanation applies. Updating the pin after an intentional correction preserves the usefulness of later runs.

The criteria also ask each project to add coverage that is not already in the suite. At launch the documented list includes Alamofire, Kitura, ObjectMapper and RxSwift, among others. The announcement's goal is thousands of projects, which is an ambition for growth. It is not a description of the initial list's size.

I value that emphasis on additional coverage. Repeating the same source patterns across many repositories provides less new information than adding a project that exercises a different use of the language. The maintainership requirement also matters because a reproducible failure becomes less useful if nobody can explain the project's assumptions or maintain its entry.

A pull request can request the comparison

Compiler contributors can request the suite by commenting on a Swift pull request. The documented instruction is short enough to keep alongside the ordinary review workflow.

@swift-ci Please test source compatibility

The suite also includes a local runner that accepts a compiler path and filters for particular repositories or actions. Its default output goes to per-action log files, with a verbose option for standard output. These controls help a contributor narrow a failing comparison to a particular project and inspect the build that failed.

The immediate benefit is a shared corpus for compatibility investigation. A language change still needs design review, and the suite's success does not establish compatibility with programs outside its index. Within that index, however, a contributor can identify the exact application revision, build action and development compiler involved. Those details make an unexpected source rejection a concrete issue that can be reproduced before a release reaches application developers.