Swift Cluster Membership separates failure detection from transport

A peer that misses a deadline may have crashed, lost a packet or simply stopped processing messages for a while. A cluster membership implementation has to turn those observations into useful information without pretending to know more than it does. Swift Cluster Membership, announced on August 27, supplies a reusable implementation of SWIM. Its design separates protocol decisions from the networking code that carries out those decisions.

The initial package is a prerelease. Its SwiftNIO transport is an example, and the announcement explicitly says that example was not implemented for production use. The reusable protocol implementation is the main focus. That distinction matters to someone choosing a component for a distributed system because a protocol engine and a complete deployed membership service have different responsibilities.

I like Swift's emphasis on local reasoning through values. Distributed failure detection necessarily deals with observations that can disagree between machines. Keeping those observations and their transitions inside an explicit protocol implementation gives the local code a defined job, even though the resulting cluster view remains weakly consistent.

One missed reply starts more investigation

The launch-day README describes the probe sequence. A member periodically chooses a peer and sends a ping. An acknowledgement establishes that the peer responded to that probe. When the response does not arrive, the origin asks other members to probe the target through ping requests.

Indirect probing gives the origin more evidence than its own connection alone. Another peer may reach a target that the origin cannot reach directly. A direct timeout consequently does not immediately become a final declaration that the target is dead. If the additional probes also fail, the protocol can mark the member suspect.

Messages also carry gossip about membership observations. The same exchanges that probe peers help spread information about other members. A node can learn about peers and status changes through those payloads, rather than contacting every member directly for every update.

The result is a weakly consistent view. Different members can have different information at a particular moment. An application using these events must account for that meaning. Treating a local membership observation as proof that every other node has the identical view adds a guarantee that this package does not provide.

For a scheduler, that distinction affects how an observation is used. Membership can inform which peers appear available. A decision that requires agreement among multiple machines needs an additional mechanism with that agreement contract. The membership library is one component of that design, and its event stream should retain its documented meaning.

The observer can be unhealthy too

The implementation includes extensions described in the Lifeguard work linked by the package. One visible mechanism is a negative acknowledgement from an intermediary. It tells the origin that the intermediary received the ping request but did not obtain the expected target response.

That information also contributes to a Local Health Multiplier affecting timeouts. The failure detector is itself running inside a process that can become slow. A missed message can say something about the observer's ability to process events as well as the target's condition. Adjusting timing according to local health gives the implementation a way to reflect that uncertainty.

This makes the timer integration part of correctness. The README says the periodic tick must not use a fixed repeating timer. The next interval is dynamic and can change as the algorithm progresses. A shell should schedule the next tick using the delay returned by the appropriate directive.

A networking adapter that copied the initial interval into a permanent timer would therefore change the algorithm's behavior. It might deliver the right messages through the right transport and still ignore the protocol's timing decisions. Runtime independence makes the adapter replaceable, but it does not make its contract optional.

Protocol callbacks return instructions

SWIM.Instance contains the runtime-independent protocol logic. A shell invokes its callbacks when a ping, response or timer event occurs. The callback returns directives describing what the shell should do next. This keeps the algorithm's decisions separate from the mechanics of channels and timers.

For example, handling a ping request can return a directive to send a ping. The directive supplies the relevant target and timing information, together with sequence numbers used by the protocol. The shell translates that instruction into an actual transport operation and later reports the resulting event to the instance.

The peer abstraction expresses another part of that boundary. Its ping operation includes a gossip payload, origin information, timeout and sequence number, with a completion callback for the response. Implementing a different transport means supplying those operations and translating incoming messages into the matching instance callbacks.

This architecture offers a useful review boundary. Changes to protocol decisions can be examined independently of a particular socket implementation. Transport code can focus on delivering messages and honoring the directives. The shell still needs correct concurrency behavior because it determines when the instance receives its observations.

Dead membership is a final state

The documentation describes eventual movement to a dead state as irrevocable removal from the cluster. It also offers an optional unreachable state, disabled by default. Those states provide different meanings, so application code should preserve their distinction instead of converting every adverse observation into one Boolean.

Suspicion expresses uncertainty after failed probes. Final removal has a stronger lifecycle meaning within the protocol. A consumer that discards the distinction loses information the failure detector maintains. The optional intermediate state can support uses needing a separate reachability judgment, but its existence does not change the default configuration.

I prefer an API that exposes those meanings directly. A type or event describing suspicion makes an application acknowledge what has actually been observed. The application can then decide which actions are reversible and which need stronger coordination before acting.

The NIO example supplies a concrete adapter

The supplied SWIMNIOShell uses a SwiftNIO datagram channel and UDP. Its channel pipeline includes a SWIM handler and an application handler receiving membership-change events. The example demonstrates how the protocol engine can fit into an existing event-driven networking model.

The instance can emit metrics through Swift Metrics and accept a Swift Log logger. Those facilities help expose the operation of the algorithm without requiring a particular transport. At launch, the project is seeking experience with the instance API before a stable release, while the NIO example remains a separate implementation with additional cleanup work identified by the maintainers.