Unhealthy pods need an explicit eviction policy

A node drain can stop behind a pod that is already failing to become ready. The application has no healthy capacity to lose from that particular pod, yet its disruption budget can still refuse eviction. That behaviour protects an opportunity for recovery, which matters when deleting the pod could destroy something the readiness condition does not describe. It also leaves a cluster administrator waiting for an application that may never recover on its own.

The January 6 explanation of unhealthy-pod eviction describes how Kubernetes 1.26 makes that choice configurable. The new spec.unhealthyPodEvictionPolicy field belongs to the PodDisruptionBudget. Application owners can retain the existing protection or allow eviction of unhealthy pods even when the application has already fallen below its availability requirement. The feature is alpha and requires an explicit API server feature gate.

I value Kubernetes for its reconciliation model. A controller can keep trying to converge after an interruption, but a policy that refuses an action remains a constraint on that convergence. This change interests me because it makes an otherwise implicit assumption about recovery visible in the workload's configuration.

Why the budget can block an unhealthy pod

The announcement describes a PodDisruptionBudget as a constraint on API-initiated eviction. Its health accounting uses status.currentHealthy and status.desiredHealthy. An eviction should not take the application's healthy count below its required count. Running pods that are unhealthy do not contribute to the healthy count, but that fact alone does not make their eviction permissible under the existing policy.

Consider an application whose budget requires two healthy pods. Suppose only one is currently healthy and another running pod cannot become ready. The failed readiness condition tells the disruption controller that the second pod does not supply healthy capacity. It does not establish whether that process is about to recover or whether deleting it would make recovery harder. Protecting it preserves that possibility while the application is already disrupted.

This explains a behaviour that otherwise looks inconsistent. A budget can refuse removal of a pod that contributes nothing to its current healthy count. Its purpose includes preserving the application's ability to regain availability. The source explicitly identifies users who depend on this behaviour to avoid data loss when a pod guards an underlying resource or storage. Readiness alone cannot express that dependency.

The same rule becomes an obstacle during maintenance. The announcement gives CrashLoopBackOff caused by a bug or misconfiguration as an example. When every pod of the application is unhealthy, eviction requests can fail against its budget and the drain cannot progress. An administrator then needs intervention beyond the normal drain operation. A conservative workload policy has become a dependency of node management.

Two policies express different recovery assumptions

IfHealthyBudget retains the existing behaviour. An unhealthy pod can be disrupted when the application has at least its desired healthy count. When the application is below that count, the budget continues to protect the unhealthy pod. This is the default policy described in the January article.

AlwaysAllow permits eviction of running unhealthy pods regardless of the application's healthy budget. The Kubernetes 1.26 API definition specifies the running phase and defines healthy pods through a Ready condition with status True. It gives node maintenance a way to proceed when a misbehaving pod would otherwise hold it up. The article describes the resulting availability as best effort. Application owners choose to accept removal of the unhealthy pod even while the rest of the application is short of healthy capacity.

For the two-pod example, the difference is concrete. With one healthy pod against a requirement of two, IfHealthyBudget preserves the running unhealthy pod. AlwaysAllow permits its eviction. Neither policy changes the fact that the application currently has only one healthy pod. They differ over whether retaining the unhealthy process is a necessary part of recovery.

The name AlwaysAllow needs to be read in the context of this field. The field governs unhealthy-pod eviction. It does not declare that every eviction of a healthy application pod is acceptable. The budget still expresses the application's requirements for voluntary disruption. Treating the setting as a general permission to disregard availability would lose the distinction the new API is designed to make.

I prefer this decision to live with the application policy. The platform administrator knows why a node needs maintenance, while the workload owner is better placed to judge whether a failed readiness condition makes a process disposable. A single cluster-wide assumption would obscure that difference.

What the alpha configuration looks like

In Kubernetes 1.26, the API server must enable PDBUnhealthyPodEvictionPolicy. The announcement specifies the argument --feature-gates=PDBUnhealthyPodEvictionPolicy=true. The manifest alone is therefore insufficient to opt into the experiment on an otherwise unchanged cluster.

The following example follows the announcement's webserver case. It assumes pods labelled app: nginx and an enabled feature gate.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: nginx-pdb
spec:
  selector:
    matchLabels:
      app: nginx
  maxUnavailable: 1
  unhealthyPodEvictionPolicy: AlwaysAllow

The selector identifies the protected pods. maxUnavailable supplies the disruption allowance, while the final field supplies the explicit treatment of unhealthy pods. Keeping those declarations together makes the recovery assumption reviewable alongside the availability policy. A reader does not have to infer it from an administrator's drain procedure.

The announcement describes API-initiated eviction as a process that triggers graceful pod termination. A direct API call, a node drain, or another cluster actor can initiate it. The new field participates in that decision. It does not repair a faulty application configuration or explain why readiness failed, so successful eviction and application recovery remain separate outcomes.

Availability includes the ability to perform maintenance

A disruption budget can protect a process whose remaining value is not represented by readiness. That is a legitimate requirement, especially for the storage-related cases the authors identify. It can also preserve a permanently broken process while preventing routine node work. Both behaviours follow from the same conservative default.

The useful operational question is whether an unhealthy pod must remain in place for the application to recover safely. If it must, IfHealthyBudget expresses that requirement. If replacement is acceptable even during an existing disruption, AlwaysAllow expresses that choice without requiring a special intervention for each drain. The alpha API makes that distinction explicit at the point where the application declares its disruption tolerance.