The Queue That Changed

Section 01

Four parts, and only one of them moved

An informer is a fixed pipeline: a Reflector holding the network connection, a FIFO of deltas, an Indexer that is the in-memory cache, and the event handlers that fire as the cache is maintained. Three of those four have been stable for years. The FIFO was replaced.

Section 02

Same input, two output shapes

Both queue types are exported, so the comparison needs no cluster and no informer — construct one of each, apply an identical sequence, then drain. The sequence interleaves two objects deliberately, because that is where the behaviours separate.

event sequence applied to BOTH queues, no Pop until fully queued:
  add alpha@rv1, add beta@rv1, update alpha@rv2, update beta@rv2, add gamma@rv1, delete gamma

DeltaFIFO  (pre-1.33 default):
  pop 1 -> 2 delta(s): [Added alpha@rv1] [Updated alpha@rv2]
  pop 2 -> 2 delta(s): [Added beta@rv1] [Updated beta@rv2]
  pop 3 -> 2 delta(s): [Added gamma@rv1] [Deleted gamma@rv1]
  TOTAL: 3 Pop(s), 6 delta(s)

RealFIFO   (default since 1.33, locked 1.36):
  pop 1 -> 1 delta(s): [Added alpha@rv1]
  pop 2 -> 1 delta(s): [Added beta@rv1]
  pop 3 -> 1 delta(s): [Updated alpha@rv2]
  pop 4 -> 1 delta(s): [Updated beta@rv2]
  pop 5 -> 1 delta(s): [Added gamma@rv1]
  pop 6 -> 1 delta(s): [Deleted gamma@rv1]
  TOTAL: 6 Pop(s), 6 delta(s)

Section 03

The deletion that vanishes

If ordinary traffic survives both queues intact, where is the loss the doc comment advertises? It is in Delete, and it is conditional. Reading the implementation first:

func (f *DeltaFIFO) Delete(obj interface{}) error {
	...
	} else {
		_, exists, err := f.knownObjects.GetByKey(id)
		_, itemsExist := f.items[id]
		if err == nil && !exists && !itemsExist {
			// Presumably, this was deleted when a relist happened.
			// Don't provide a second report of the same deletion.
			return nil
		}
	}

RealFIFO’s equivalent has no guard at all — it appends the delta unconditionally:

func (f *RealFIFO) Delete(obj interface{}) error {
	f.lock.Lock()
	defer f.lock.Unlock()

	f.populated = true
	f.checkSynced_locked()
	retErr := f.addToItems_locked(Deleted, false, obj)

	return retErr
}

Running both cases makes the divergence concrete.

CASE 1 — Delete() an object that is in neither the queue nor knownObjects
         (what a relist-while-deleted race produces)
  DeltaFIFO yields: []
  RealFIFO  yields: [Deleted ghost]

CASE 2 — Add() then Delete() twice (duplicate deletion report)
  DeltaFIFO yields: [Added twice Deleted twice]
  RealFIFO  yields: [Added twice Deleted twice Deleted twice]

Section 04

The version everyone cites is off by three

Secondary sources place this change at 1.36. The selection logic reads a gate, and the gate registry is unambiguous — it is worth reading rather than trusting a summary.

InOrderInformers: {
	{Version: version.MustParse("1.33"), Default: true, PreRelease: Beta},
	{Version: version.MustParse("1.36"), Default: true, PreRelease: GA, LockToDefault: true},
},

Default-on since 1.33. What 1.36 added was LockToDefault. That is not a documentation nicety — it is executable, and it can be tested.

$ go run ./02-informers/demos/gate-locked
KUBE_FEATURE_InOrderInformers env = ""
InOrderInformers enabled = true

$ KUBE_FEATURE_InOrderInformers=false go run ./02-informers/demos/gate-locked
KUBE_FEATURE_InOrderInformers env = "false"
E0911 16:54:16.390789  envvar.go:179] "Could not set feature gate, feature is locked"
  feature="InOrderInformers" desiredState="false" lockedState=true
InOrderInformers enabled = true

Section 05

Store before notify, measured

Chapter 01 rested on a guarantee: when a handler runs, the cache already holds at least the version that triggered it. That is what makes it safe for the handler to throw the object away and enqueue only a key. The ordering is visible in processDeltas — store mutation precedes the callback in every branch — but it is worth confirming empirically rather than by reading.

case Sync, Replaced, Added, Updated:
	if old, exists, err := clientState.Get(obj); err == nil && exists {
		if err := clientState.Update(obj); err != nil { return err }
		handler.OnUpdate(old, obj)
	} else {
		if err := clientState.Add(obj); err != nil { return err }
		handler.OnAdd(obj, isInInitialList)
	}

handler invocations         : 101
cache HIT at handler time   : 101
cache MISS at handler time  : 0
cache STALE (data older than evt): 0   <- must be 0
cache already FRESHER (ahead)   : 0

-- proof the comparison has teeth --
events carrying a parsed value > 0 : 100
highest value seen by a handler    : 100

Section 06

The list step is now a stream

One more change sits upstream of the FIFO. The Reflector no longer necessarily performs a paged LIST to prime the cache; it can open a watch stream that replays initial state as synthetic events, then signals completion with a bookmark.

// useWatchList if turned on instructs the reflector to open a stream to bring data
// from the API server.
useWatchList bool
...
r.useWatchList = clientfeatures.FeatureGates().Enabled(clientfeatures.WatchListClient)
if r.useWatchList && watchlist.DoesClientNotSupportWatchListSemantics(lw) {
	... r.useWatchList = false      // automatic fallback
}

The gate flipped default recently: WatchListClient was Beta and off at 1.30, and Beta and on from 1.35. The enhancement is KEP-3157, whose title is the clearest one-line description of the feature: “Allow informers for getting a stream of data instead of chunking.” There is an automatic fallback when the server does not support watch-list semantics, so this is not a hard compatibility break.

Section 07

Further reading

All links returned HTTP 200 on 2026-09-11 unless noted.

Section 08

Closing note — what varies, and what was not verified

Durable versus run-specific

• Durable: the pop counts (3 vs 6), the grouping behaviour, the empty result in Case 1, the double-delete asymmetry, and the locked-gate refusal. These are deterministic — they involve no timing and no goroutines, and should reproduce byte-for-byte on client-go v0.37.0.
• Run-specific: the timestamp in the gate error log, and the 101 handler invocations in §05 (101 rather than 100 because the initial Add from cache sync counts).

What was not verified

• The batched delta path. processDeltasInBatch and its TransactionStore were read but not exercised. Everything in §05 covers the non-batched path.
• The streaming list. See §06 — fallback was taken in every run here.
• Which release first contained the_real_fifo.go. The file header says 2025 and the gate says 1.33, which are consistent, but the first tagged release containing the file was not established.
• Real resourceVersion semantics. The fake clientset leaves resourceVersion empty on every object, so nothing here exercises version-based comparison, conflict detection, or watch resumption. The §05 ordering result rests on payload ordering instead.
• Atomic events. AtomicFIFO and UnlockWhileProcessingFIFO are Beta as of 1.36 and were not exercised; RealFIFOOptions encodes constraints between them (atomic events require KnownObjects == nil) that were read, not tested.

A note on the comparison’s fairness

Both queues were given a KnownObjects store that was kept in sync by the driver, which is what an informer does. Constructing DeltaFIFO with knownObjects == nil takes a different branch in Delete that drops the notification under a different condition. The Case 1 result is specific to the configuration an informer actually uses.

Section 09

Spoken drills