Kubernetes 1.28.2 repairs concurrent scheduler event tracking

A pending pod can remain unscheduled even after a relevant cluster change if the scheduler loses track of that change during its current attempt. Kubernetes 1.28.2 corrects such a regression. A pod experiencing concurrent events could enter the unschedulable queue incorrectly and remain there until the next periodic purge, up to five minutes later, if no other event arrived for it.

The September 13 maintenance release includes a scheduler event-tracking correction that explains the mechanism in detail. The implementation had used an existing list element to mark which events occurred while a pod was in flight. Removing that element could disconnect the marker from later events, leaving the scheduler with an incomplete history at the moment it chose where to queue the pod.

I value reconciliation that can recover after missed notifications. This bug shows why a recovery mechanism and a prompt response are different properties. A periodic purge can eventually revisit a pod, while the loss of a concurrent event still creates an avoidable scheduling delay.

A scheduling attempt needs a record of concurrent changes

The patch defines in-flight pods as pods already removed from the scheduling queue whose processing has not yet finished. That includes processing during scheduling, permit handling, or binding. Cluster events can arrive while those operations are under way. The scheduler needs to consider those events when deciding whether a failed attempt should be retried.

The old implementation maintained a received-event list and associated an in-flight pod with a previous event. That event acted as a marker. Looking at the elements after it was supposed to identify changes that arrived after the pod's attempt began. This avoided needing an independent event history for each pod.

The assumption failed when the marker itself left the list. An event could be removed once the pods that were in flight when it arrived had finished. Another pod could still be using that element as its starting point. Calling the removed element's Next method then returned no next element, even if later concurrent events existed in the retained list.

The resulting empty traversal was false evidence about the history. It did not mean no event occurred while the pod was being processed. It meant the position used to discover those events no longer participated in the list. The scheduler could therefore decide against a prompt retry using an incomplete view of concurrent activity.

Each pod now has its own position in the list

The correction replaces that arrangement with an inFlightEvents list containing both cluster events and in-flight pod entries. Each pod receives its own entry when its scheduling attempt begins. The scheduler can then traverse entries after that pod's marker to examine subsequent events.

The implementation comments say entries are appended while holding a mutex, which serialises their order. A pod entry records the pod as it existed when the scheduling attempt started. The map of in-flight pods points to these list elements, connecting each active attempt to an explicit position in the shared sequence.

When a scheduling attempt finishes, its pod entry can be removed after the intervening events have been considered. Events at the front of the list can then be discarded when no remaining in-flight pod needs them. The comments explain that other active attempts began later, so older events preceding those attempts are no longer necessary for their histories.

This preserves the advantage of a shared event list while making marker lifetime explicit. The pod's marker remains present for the duration of the attempt that owns it. Event cleanup no longer has to preserve an unrelated event merely because another attempt uses that event as an implicit boundary.

The retry decision depends on incomplete knowledge too

One branch in the patch handles pods without a recorded unschedulable plugin. In that case, the scheduler cannot confidently identify which event might make the pod schedulable. The revised code looks for any concurrent event and sends the pod through backoff when one exists, because the event may be relevant.

That is a conservative response to limited information. It does not claim every event makes the pod schedulable. It preserves the opportunity for another attempt when the scheduler cannot rule out a relevant change. Losing the concurrent history would defeat that policy by making a potentially relevant event appear absent.

The patch also revises the queue's tests alongside the implementation. Its subject is tracking concurrent events, and the code comments make the ordering and cleanup rules explicit. Those rules are the useful part of the maintenance change for readers writing other asynchronous controllers. A marker needs a lifetime tied to the operation using it.

I find the direct explanation of the removed-list-element failure useful because it identifies a precise invariant. The event list must retain a traversable boundary for every active attempt. A periodic recovery loop can limit the eventual delay, but it cannot substitute for preserving that invariant during ordinary scheduling.

Kubernetes 1.28.2 repairs the regression without claiming that every pending pod should schedule immediately. Resource constraints and other scheduling conditions still need to be satisfied. The specific correction ensures that the queue decision can consider events received during the attempt instead of overlooking them because its marker was removed.