Kubernetes 1.5 separates stateful identity from replaceable processes

An unreachable node leaves a cluster with incomplete information. The machines observing it cannot necessarily tell whether its processes have stopped. Creating another process with the same identity can therefore be a dangerous response, especially when that identity belongs to a database member. Kubernetes 1.5 makes this ambiguity central to its support for stateful workloads. Its node controller stops force deleting pods on unresponsive nodes, a change explained in the December 13 release announcement.

StatefulSet, previously named PetSet, reaches beta in the same release. It supports workloads that need persistent identity or storage associated with a particular instance. The important connection is between that identity and the decision to replace its process. A controller cannot safely treat an inaccessible machine as proof that an old identity has ceased to exist. The operational consequence is that recovery may wait for evidence or an administrator's decision.

I find this a useful expression of the limits of cluster automation. Declaring the intended workload does not give the controller knowledge that the network has withheld. A stateful workload needs rules for what happens while that knowledge is missing, and those rules affect availability as well as data safety.

Failure detection cannot prove termination

The announcement identifies several ways the old pod can be confirmed dead. A returning kubelet can report that the process terminated. An administrator can delete the node, or a database administrator can force delete the pod after deciding that replacement is safe. The command line now warns users who request force deletion. These paths place responsibility for the missing information with the participant able to establish it.

Consider a database member that remains active on a node whose connection to the cluster control plane has failed. The application may still have connections that the control plane cannot see. If another process starts under the same logical identity, the application has to contend with that duplicate. The release's decision to retain the old pod record makes this uncertainty visible instead of allowing the controller to proceed as though termination were established.

This behavior has an availability cost. A replacement can remain blocked while the operator determines what happened to the old instance. That delay is an expected consequence of the safety rule. It also means that a runbook which treats force deletion as routine cleanup can undo the protection the new behavior provides. The relevant decision is whether the old member can still act, which requires knowledge beyond a missing heartbeat.

The change is also recorded in the source tree's changelog at the 1.5.0 tag, which points to the node-controller change among actions requiring attention. This is an upgrade behavior to explain to administrators before they encounter an unreachable machine. Retaining an object is not evidence that the controller has forgotten how to replace a failed workload.

Planned disruption has a different decision point

PodDisruptionBudget also reaches beta in 1.5. It lets an application owner specify the minimum number or percentage of selected replicas that must remain available when a voluntary eviction occurs. The release announcement says that the drain command now honors this budget. That establishes a connection between an application's availability requirement and the cluster administrator's maintenance operation.

The distinction between planned maintenance and an involuntary failure is essential. A maintenance tool can request permission before removing a pod. Hardware failures cannot ask the API whether they are allowed to stop. The budget therefore constrains actions that cooperate with the eviction mechanism, including the maintenance path described in this release. It does not make the selected workload immune to hardware failure or establish that its data is correct.

The versioned policy API definition makes the decision concrete. Its specification contains minAvailable and a pod label selector. An eviction is allowed when the remaining selected pods still meet the availability requirement. A small example for a workload whose pods carry the label app: ledger is:

apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
  name: ledger-maintenance
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: ledger

This object expresses a requirement for at least two available selected pods after a permitted eviction. It does not create those pods, configure database membership or establish quorum by itself. The application owner must choose the number from the application's actual requirements. For a hypothetical three-member system requiring two functioning members, the budget represents an appropriate maintenance constraint while all members are healthy.

An existing unavailable member changes that calculation. Removing another healthy member can violate the minimum even if the administrator only intends a short interruption. Blocking that eviction gives the maintenance operation a reason to wait. The application owner no longer needs the person draining the node to remember an informal rule about that particular workload.

Status explains why an eviction must wait

The policy types include more than the requested minimum. Status records current healthy pods, desired healthy pods and the number of disruptions currently allowed. It also records evictions the API server has processed but the budget controller has not yet observed as pod deletion. This is a distributed operation with an interval between accepting a request and observing its effect.

That interval matters when more than one maintenance request arrives. A decision based only on a count of pods observed at one instant could overlook an eviction already accepted elsewhere. The disruptedPods accounting represents work in progress so that a pending deletion is relevant to the next decision. The type definition also notes that status can trail the actual system and ties the validity of its information to the observed object generation.

These details help explain why availability policy belongs in a shared API. Separate maintenance scripts need a common place to account for accepted disruption. An application-specific note in a runbook cannot coordinate requests from different actors. The budget provides a machine-readable requirement and status that expose the relationship between the selected workload and the maintenance action.

I like this division of responsibility. The application owner supplies the availability requirement, and the administrator requests an operation through a mechanism that can respect it. Neither person has to encode the other's entire procedure, although both still need to understand what the minimum means.

Runtime choice remains experimental

The same release introduces an alpha Container Runtime Interface that abstracts the runtime from the kubelet. The announcement describes an experimental Docker integration for testing. Its maturity differs from the beta stateful workload features, so treating every 1.5 addition as equally ready for routine operation would misrepresent the release.

The architectural purpose is to give runtime implementations a defined integration point. That can reduce the extent to which runtime-specific behavior is embedded in kubelet integration, but the alpha status still matters to anyone evaluating it. Runtime interchangeability is an engineering objective whose practical limits need to be established by implementation and tests. Introducing an interface does not make every implementation equivalent.

Node conformance tests, also described in the announcement, address a related question. The containerized framework checks whether node software meets minimum requirements for joining a cluster. This gives operators a way to examine the node environment independently of an application's behavior. Passing such a check does not establish that a database has a safe recovery procedure, because that procedure depends on the database and its deployment.

The workload still owns its data model

StatefulSet supplies instance identity and associated storage semantics, while a disruption budget constrains cooperative maintenance. Together they make Kubernetes more useful for stateful applications, but the database still defines how members agree and what safe replacement means. The node-controller change is unusually explicit about where that application knowledge enters an otherwise automated system.

For someone evaluating a distributed database on Kubernetes, the practical question is how its recovery procedure interacts with retained pod identities. Waiting for a kubelet to return and declaring an old node permanently gone are different operational decisions. Both can permit progress, but they rely on different evidence. A procedure that documents that evidence gives the administrator a basis for acting when automatic recovery pauses.

My preference is for a controller that preserves an unresolved identity until there is a defensible replacement decision. The additional maintenance policy is useful for the same reason. It lets the cluster represent an application's stated availability requirement at the point where an administrator requests disruption, with status that explains whether the request can proceed.