The Cache You Read From

Section 01

Counting the API calls

The Lister is a read-only typed accessor over the Indexer. In controller-runtime the same role is played by the client the manager hands you, which is why many people have never typed the word Lister and use one on every line of their reconciler. The claim worth testing is not that this is faster — it is that it does not talk to the API server at all.

The fake clientset records every action it receives, which makes that directly countable.

demos/cache-vs-apiserver · 2000 reads each wayrung 1 · measured

API actions after informer startup, settled: 2
   list configmaps
   watch configmaps

                                reads    API actions     total time
Lister (local cache)             2000              0          309µs
typed client (API server)        2000           2000         4.07ms

per-read: lister 154ns   client 2.035µs   ratio 13x

go run ./03-listers/demos/cache-vs-apiserver

Finding

An informer costs exactly two API calls for its entire lifetime: one list to prime the cache and one watch to keep it current. Every subsequent read — 2000 of them here, and millions over a controller's life — costs nothing on the wire.

That is the whole reason a cluster with hundreds of controllers does not melt its own control plane. It is also why MaxConcurrentReconciles is safe to raise: more workers means more cache reads, not more API load.

Do not quote the 13×

Both sides of that ratio ran against an in-process fake. The “API server” here is a map behind a mutex — no network, no TLS, no authentication, no etcd. A real API server read costs milliseconds, not two microseconds, so the true ratio is larger by three or four orders of magnitude. The 0 versus 2000 is the durable finding; the timing ratio drastically understates reality and should not be repeated as a number.

The Reflector pays once so every reader pays nothing. The upper path terminates in memory — the crossed arrow is the API call that never happens. The lower path is what the same code costs if you reach for the typed client out of habit, multiplied by every worker in every controller in the cluster.

Section 02

Writes bypass the cache entirely

Reads are served locally. Writes are not — they go straight to the API server, and the only path by which a write becomes visible to your own cache is the watch stream coming back. That is a loop through the network, not a local update.

Measuring the latency of that loop against a fake clientset is useless, because the fake delivers watch events in microseconds; an attempt to measure read-after-write staleness this way returned 0% stale across 200 trials, which says everything about the test double and nothing about Kubernetes. The structural fact is better shown deterministically: cut the watch, then write.

demos/stale-cache · watch stopped, then one writerung 1 · measured

after initial sync                 cache=original   api-server=original   AGREE

-> stopping the watch (simulates a dropped connection / partition)
-> Update() returned successfully

t+0s after the write               cache=original   api-server=updated    *** DIVERGED ***
t+100ms after the write            cache=original   api-server=updated    *** DIVERGED ***
t+1s after the write               cache=original   api-server=updated    *** DIVERGED ***

go run ./03-listers/demos/stale-cache

Two separate circuits. The write leaves and never comes back except by the dashed route. Cut that route and the reconciler keeps reading original forever while the cluster holds updated — the write did not fail, and no error was returned anywhere.

What this does and does not model

A real Reflector would notice the broken watch and relist, repairing the divergence — that recovery is the subject of chapter 02. This demo removes the recovery deliberately to isolate the structural point. What it models faithfully is the direction of data flow; what it does not model is how long a real cluster leaves you diverged, which the fake is far too fast to show.

Section 03

The resync that does not resync

Most engineers carry a belief that the periodic resync repairs a cache that has drifted from the server. The source says otherwise in a single sentence, and it is worth reading to the end of the comment before trusting the mechanism.

controller-runtime v0.25.0 · pkg/cache/cache.go · on SyncPeriodrung 2 · read the source

// SyncPeriod will locally trigger an artificial Update event with the same
// object in both ObjectOld and ObjectNew for everything that is in the
// cache.
//
// Predicates or Handlers that expect ObjectOld and ObjectNew to be different
// (such as GenerationChangedPredicate) will filter out this event, preventing
// it from triggering a reconciliation.
// SyncPeriod does not sync between the local cache and the server.

“The same object in both” is a strong claim, and it is testable. Running an informer with a resync period and comparing the two arguments handed to every OnUpdate:

demos/resync-replay · 1 s resync, one genuine change, 3.5 s observationrung 1 · measured

resync period                      : 1s
OnUpdate invocations               : 4
  ... with a genuine payload change: 1
  ... with old payload == new      : 3   <- resync replays
  ... where old and new are the SAME pointer: 3

go run ./03-listers/demos/resync-replay

Finding

Not merely equal — the same pointer. Three of the four callbacks received one object address in both ObjectOld and ObjectNew. There is no version comparison a handler could make that would distinguish them, because there are not two objects.

A resync is a replay of what the cache already holds. It issues no request, so it cannot discover an object the cache never learned about, and it cannot repair a value that drifted. Its actual purpose, per the same doc comment, is narrow: insurance against a bug in the controller or in controller-runtime that drops a requeue.

The loop closes inside the process. The dashed arc is the entire resync mechanism: read the Indexer, hand each object to the handlers twice. The crossed link on the right is the request it never makes — which is why it is not, and cannot be, a defence against a stale cache.

Section 04

The predicate that eats your backstop

The doc comment names the consequence explicitly, and it lands on the single most commonly added predicate in controller-runtime. GenerationChangedPredicate exists to suppress reconciles caused by status-only writes. A resync event, having identical old and new, is indistinguishable from a no-op — so it is dropped.

demos/resync-replay · calling the predicate directlyrung 1 · measured

GenerationChangedPredicate.Update(resync event, old==new) = false   <- filtered out
GenerationChangedPredicate.Update(real change, gen 7->8)  = true

Finding — the combination that bites

Each of these is individually reasonable. Adding GenerationChangedPredicate to avoid status-write reconcile storms is standard advice. Lowering SyncPeriod from ten hours to something shorter, believing it a safety net, is common. Together, the safety net is removed entirely and nothing reports it: the resync fires on schedule, the predicate discards it, and the reconciler is never called.

The failure is silent in the worst way — the timer keeps firing, so any metric counting resync events still moves.

The backstop and the filter are on the same wire. The predicate cannot tell a resync from a no-op write, because by construction there is nothing to tell apart. Every resync takes the upper branch.

Section 05

What upstream says to do instead

The same doc comment that removes the illusion also states the remedy, and it is not to lower the global period.

controller-runtime v0.25.0 · pkg/cache/cache.go · on SyncPeriodrung 2 · read the source

// If you want
// 1. to insure against missed watch events, or
// 2. to poll services that cannot be watched,
// then we recommend that, instead of changing the default period, the
// controller requeue, with a constant duration `t`, whenever the controller
// is "done" with an object, and would otherwise not requeue it, i.e., we
// recommend the `Reconcile` function return `reconcile.Result{RequeueAfter: t}`,
// instead of `reconcile.Result{}`.

The difference is not cosmetic. SyncPeriod is one global timer that replays the cache through the predicate chain. RequeueAfter is a per-object schedule that puts a key on the workqueue directly, downstream of every predicate, and therefore cannot be filtered out by one.

Entry point decides survivability. SyncPeriod enters as an event and must survive the predicate chain. RequeueAfter enters as a key already past it. That is why the recommendation is a per-reconciler return value rather than a manager-wide setting.

Section 06

Closing note — what varies, and what was not verified

Durable versus run-specific

Durable: 2 API actions for informer startup; 0 actions for any number of Lister reads; the cache/server divergence persisting indefinitely once the watch is cut; resync passing the same pointer as both arguments; GenerationChangedPredicate returning false on a resync event.
Run-specific: all timings (309 µs, 4.07 ms, the 13× ratio) and the exact count of resync callbacks observed in a fixed sleep window — 4 here, but a timing race with the 1 s resync tick.

Traps this chapter fell into

A miscounted API action. The first run of cache-vs-apiserver attributed 1 action to the Lister loop. Investigating showed it was the informer's watch registering asynchronously after WaitForCacheSync returned. A 250 ms settle before baselining makes it a clean 0, and the demo now prints any unexpected action rather than hiding it.
A measurement that was too good. Read-after-write staleness measured 0% across 200 trials with a p50 catch-up of 1 µs. That is the fake clientset being in-process, not Kubernetes being strongly consistent. The claim was demoted rather than published, and §02 uses the deterministic cut-watch demo instead.
A resync period that was silently changed. Asking for 300 ms produced "Warning: resync period is too small. Changing it to the minimum allowed value" minimumResyncPeriod="1s". The demo now requests 1 s so the number on the page is the number in effect.

Not verified

Real staleness latency. Not measurable with this harness — see above. The kubernetes.io staleness post is cited as rung 3 for the existence of the window, not for any duration.
controller-runtime's own client. §01 measures a client-go Lister. The controller-runtime cache-backed client documents the same split (Options.Cache.Reader, DisableFor, Unstructured) and was read, but the API-action count was not repeated through it.
EnableReadYourWritesConsistency. Documented as defaulting to false and requiring the cache reader to implement cacheapi.Informers; read, not exercised.
The 10-hour default in practice. The constant defaultSyncPeriod = 10 * time.Hour and its 10% jitter were read from source; no run here waited ten hours.

Section 08

Spoken drills

Your reconciler calls Get forty times per invocation and runs at 50 reconciles a second. What is that costing the API server?

A strong answer hits

Nothing, if those are cache reads — measured 0 API actions for 2000 reads
An informer costs 2 calls total: one list, one watch, for the process lifetime
The cost that does scale is memory, not API traffic — cache size and index count
The honest half: it is zero only while the object type is cached. A type on DisableFor, or an unstructured read with unstructured caching off, goes to the wire every time — and that is invisible at the call site

check against §01

A teammate lowers SyncPeriod to 10 minutes “so we recover faster from missed events.” What do you tell them?

A strong answer hits

Resync does not contact the API server — it cannot recover a missed event or a drifted value
It replays the cache as Update with the same pointer on both sides
If the controller uses GenerationChangedPredicate, every one of those is filtered out
The upstream recommendation is reconcile.Result{RequeueAfter: t} per object instead
The honest half: resync is not useless — it genuinely insures against a dropped requeue caused by a bug. It is just insurance against your bugs, not against the network

check against §03, §04 and §05

You create an object and immediately list to confirm it. It is not there. Walk through what happened.

A strong answer hits

The write went to the API server directly; the read came from the local cache
The only route back into the cache is the watch stream — a network round trip
Not a bug and not an error; nothing failed
Level-triggering makes it survivable: the watch event will arrive and re-enqueue the key
The honest half: it becomes a real bug if you treat absence as “should create” and create twice, or as “should delete” and delete something live. EnableReadYourWritesConsistency exists for this, and defaults to off

check against §02

Why is RequeueAfter structurally different from lowering the sync period, rather than just a finer-grained version of it?

A strong answer hits

Different entry point: SyncPeriod enters as an event, upstream of predicates
RequeueAfter enqueues a key directly, downstream of every predicate
So it cannot be filtered out by a predicate, and it is per-object rather than global
The honest half: it is also strictly more expensive — you are choosing to reconcile that object forever on a timer, so the interval is a real load decision, not a free safety margin

check against §05

Counting the API calls

Writes bypass the cache entirely

The resync that does not resync

The predicate that eats your backstop

What upstream says to do instead

Further reading

Closing note — what varies, and what was not verified

Durable versus run-specific

Traps this chapter fell into

Not verified

Spoken drills