Section 01
The word upstream wrote in capitals
Open the reconcile package in controller-runtime and read the doc comment on Request. Three lines in there is a word set in capitals — not bold, not italics, capitals — and it is the whole design in one sentence.
// Request contains the information necessary to reconcile a Kubernetes object. This includes the // information to uniquely identify the object - its Name and Namespace. It does NOT contain information about // any specific Event or the object contents itself.
That is a constraint, not a description. The handler that turns an event into queue work is EnqueueRequestForObject, and you can watch it perform the discard: it reads two fields off the event's object and drops everything else on the floor.
func (e *TypedEnqueueRequestForObject[T]) Create(ctx context.Context, evt event.TypedCreateEvent[T], q workqueue.TypedRateLimitingInterface[reconcile.Request]) {
if isNil(evt.Object) { ... }
item := reconcile.Request{NamespacedName: types.NamespacedName{
Name: evt.Object.GetName(),
Namespace: evt.Object.GetNamespace(),
}}
addToQueueCreate(q, evt, item)
}
Section 02
What 200 writes actually cost
The claim is testable without a cluster. A fake clientset, a real SharedInformer, a real workqueue, and one worker whose reconcile takes 20 ms. Write to one ConfigMap 200 times, 1 ms apart, and record which values the reconciler actually observed.
writes issued : 200 over 458ms events delivered : 200 keys Add()ed : 200 reconciles executed : 24 collapse ratio : 8.3x values observed : [1 10 19 27 36 45 54 62 71 80 89 97 106 115 124 132 141 150 159 168 176 185 194 200] final value in cache : 200 intermediate values never observed: 176 of 200
Every event was delivered. Every key was added. The queue absorbed the difference. Note the observed list: the reconciler saw 1, then 10, then 19 — it never once saw 2 through 9, and it did not need to. Each reconcile read whatever was current at the moment it ran.
The reconciler skipped 88% of the states the object passed through and still finished correct. An edge-triggered controller consuming deltas would have had to process all 200 — and would have been wrong if it processed 199 of them, because the arithmetic of "apply each delta" has no tolerance for a gap.
Section 03
Collapse is load-dependent, not a constant
The obvious follow-up: is 8.3× a property of the design, or of this particular load? Sweeping the reconcile cost while holding the write burst fixed answers it, and the answer contains a crossover.
cost writes reconciles collapse skipped 0s 200 200 1.0x 0 1ms 200 200 1.0x 0 5ms 200 90 2.2x 110 10ms 200 46 4.3x 154 20ms 200 24 8.3x 176 50ms 200 11 18.2x 189 100ms 200 6 33.3x 194
Below the crossover there is no deduplication at all. At 0 ms and 1 ms reconcile cost the controller performed exactly 200 reconciles for 200 events, a 1.0× ratio, zero states skipped. A level-triggered controller under light load behaves identically to an edge-triggered one.
Collapse is not a feature the workqueue applies; it is what happens when the queue is asked to hold more than the worker can drain. The mechanism only engages when you are falling behind — which is exactly when you need it, and exactly when an edge-triggered design would be accumulating an unbounded backlog instead.
The x-axis uses ordinal spacing, not a linear or log scale — the sampled costs are 0, 1, 5, 10, 20, 50 and 100 ms and are drawn evenly. The shape of the curve between measured points is therefore not meaningful; only the seven points are.
Section 04
The queue converts an arrival rate into a service rate
If collapse depends on load, what precisely sets the reconcile count? Two more sweeps isolate it. First, hold cost and spacing fixed and vary how many writes arrive.
cost writes reconciles collapse skipped 20ms 100 13 7.7x 87 20ms 200 24 8.3x 176 20ms 400 48 8.3x 352 20ms 800 91 8.8x 709
Reconciles scale linearly with the write count — because at fixed spacing, more writes means a longer burst. The collapse ratio barely moves. Now hold the count fixed and change the spacing instead, which changes the burst's wall-clock duration without changing how many events occur.
gap=0ms 20ms 200 13 15.4x 187 gap=1ms 20ms 200 24 8.3x 176 gap=4ms 20ms 200 58 3.4x 142 gap=8ms 20ms 200 106 1.9x 94
Re-running the gap=0ms case in full shows 198 events delivered for 200 writes — the fake clientset's watch buffer dropped two. That is a limitation of the test double, not of Kubernetes. But it is also an unplanned demonstration of the chapter's thesis: two events vanished, nothing noticed, and the controller still converged on 200. Under an edge-triggered design those two lost deltas would have left the final state permanently wrong by exactly their contents.
Reconcile count tracks the wall-clock duration of the burst divided by the cost of one reconcile — not the number of events. 200 events compressed into 0 ms produced 13 reconciles; the same 200 events spread over 8 ms apart produced 106. The event count was identical in both runs.
That is the real service the workqueue provides: it decouples the rate at which the world changes from the rate at which your controller does work. Your controller's load is bounded by its own service rate, and an arbitrarily hostile write storm cannot raise it.
Section 05
The bill an edge-triggered design would have run up
The counterfactual is worth drawing precisely, because the losses are not a single missing feature. Queueing payloads instead of keys removes four separate properties at once, and each removal is silent.
The restart row is the one that matters most in production and is hardest to test for. A key-based controller that restarts lists everything, enqueues everything, and converges. There is no equivalent recovery for a controller that consumed deltas, because the transitions it missed were never stored anywhere.
Section 06
Where the rule is written down
This is not a convention that emerged from the code. It is a stated architectural principle, and the phrasing of the second sentence is what makes it binding.
Functionality must be *level-based*, meaning the system must operate correctly given the desired state and the current/observed state, regardless of how many intermediate state updates may have been missed. Edge-triggered behavior must be just an optimization.
Watches are the optimisation. They are not the mechanism, and a design that cannot survive losing them is not level-based. The practical restatement in the controller-writing guide names the exact trap:
Level driven, not edge driven. Just like having a shell script that isn't running all the time, your controller may be off for an indeterminate amount of time before running again. If an API object appears with a marker value of `true`, you can't count on having seen it turn from `false` to `true`, only that you now observe it being `true`. Even an API watch suffers from this problem, so be sure that you're not counting on seeing a change unless your controller is also marking the information it last made the decision on in the object's status.
That closing clause is the escape hatch for the rare case where you genuinely need to know a transition happened: record what you last acted on in the object's own status, and compare against it. The transition becomes derivable from two levels rather than from an event you hope you received.
Section 07
Further reading
Every link below was requested and returned HTTP 200 on 2026-09-11, except where noted.
Section 08
Closing note — what varies, and what was not verified
Durable versus run-specific
- Durable: that collapse is 1.0× when reconcile cost is below the inter-arrival interval; that reconcile count tracks burst duration ÷ reconcile cost; that the final observed value equals the final written value.
- Run-specific: every absolute number here — 24 reconciles, 8.3×, the 458 ms burst window, and the exact observed-value list. These depend on scheduler timing and will differ on your machine and between runs on mine.
What this demo is not
The informer here is driven by a fake clientset, not an API server. That is deliberate — it makes the queueing behaviour observable without cluster noise — but it means the measurement covers the client-side pipeline only. Nothing here measures watch latency, etcd, or API-server-side behaviour, and the write-to-event delay is far shorter than any real cluster's.
Not verified
- The 20 ms reconcile cost is a
time.Sleep, which models an I/O-bound reconciler. A CPU-bound reconciler competing for the same cores would behave differently and was not tested. - Single worker throughout.
MaxConcurrentReconciles > 1changes the service rate and was not swept; the per-key serialisation guarantee that makes it safe is measured in chapter 04, not here. - The
principles.mdtext is quoted from the archived design proposals. Its original authorship date is not established here beyond the archive's own history.
Section 09
Spoken drills
Say these out loud. Score yourself on how many beats you hit before reading the list — not on wording.
Your controller is level-triggered. Under what conditions does that buy you nothing at all?
A strong answer hits
- When the worker keeps up — reconcile cost below the inter-arrival interval
- Measured: 1.0× collapse, zero states skipped, at 0 ms and 1 ms cost
- The dedup path never engages because the key is never dirty-and-processing at once
- The honest half: it still buys crash recovery and order-independence, which are not load-dependent — only the collapse is
check against §03
A reviewer says “we should put the changed object in the queue so Reconcile doesn’t have to re-read it — it’s a free optimisation.” Respond.
A strong answer hits
- Entries stop comparing equal, so deduplication silently stops
- Two entries for one object can be held by two workers — the single-writer guarantee goes
- Ordering becomes load-bearing; restart can no longer recover, because transitions are not stored anywhere
- The read was not expensive anyway — it hits the local indexer, not the API server
- The honest half: the reviewer is right that it saves a read; the answer is that the read is nearly free and the four properties are not
check against §01 and §05
200 update events hit one object during a 30-second reconcile. How many reconciles follow, and what actually determines that number?
A strong answer hits
- Exactly one follow-up reconcile for that key
- The number is set by burst duration ÷ reconcile cost, not by event count
- Measured: 200 events over 0 ms gave 13 reconciles; the same 200 over a longer window gave 106
- The honest half: “one” is the per-key answer; across many keys the queue is doing this independently per key, and total throughput is bounded by worker count
check against §02 and §04
Your controller must act when a field flips from false to true. Upstream says you cannot rely on seeing the transition. How do you build it?
A strong answer hits
- You cannot observe edges reliably — even a watch misses them across restarts and relists
- Record what you last acted on in the object’s own status
- Derive the transition by comparing current level against that recorded level
- This is the same reasoning behind
observedGeneration - The honest half: this makes the transition durable but costs you a status write, and that write generates another event — so the reconciler has to be idempotent about its own writes
check against §06