The Event-Driven Machine  ·  Chapter 00  ·  start here

The Whole Machine

Every Kubernetes operator is built from the same four parts, wired the same way, for the same reason. This page explains what each part is and why it exists — in pictures, in plain language, with no code. If you have ever written a Reconcile function without being sure what called it or why, start here. The five chapters that follow take each part apart and measure it; this one just shows you the shape.

orientation chapter — no new measurements
every number on this page is carried from a measured run in chapters 01–05, which show the workings
concepts current for Kubernetes v1.37 · client-go v0.37.0 · controller-runtime v0.25.0

Section 01

What a controller is actually for

You tell Kubernetes what you want: three copies of this application, a certificate for that domain, a database of this size. That is the desired state. Separately, there is whatever is actually running right now — the actual state. The two drift apart constantly, because machines fail, people edit things, and networks break.

A controller is a program with one job: notice the gap, and close it. Then keep doing that, forever. Everything else in this page exists to make that job cheap, safe and reliable.

DESIRED — what you asked for replicas: 3 ACTUAL — what is running 2 pods alive missing the controller compare, then close the gap reads reads acts: start one more pod … then check again, forever
This loop is the entire job. Read what was asked for, read what exists, do whatever closes the difference, repeat. A controller never finishes — it is a loop that runs for the lifetime of the cluster.

Section 02

The one idea: react to the state, not to the news

Here is the decision everything else follows from. When something changes, Kubernetes sends your controller a notification. The tempting design is to act on that notification — "replicas went from 3 to 5, so start 2 more". Kubernetes does the opposite. The notification is treated as nothing more than a nudge meaning "go and look again". Your controller then reads the current state from scratch and works out what to do from that.

The two styles have names borrowed from electronics. Acting on the change itself is edge-triggered. Acting on the current value is level-triggered. Kubernetes is level-triggered, and this is written down as an architectural rule rather than left to taste.

the value in the cluster, over time low high EDGE-TRIGGERED — act on the change this one is lost acts acts one missed message and it is wrong from here on LEVEL-TRIGGERED — read the current value looks at what is true now, every time — a lost message just means it looks again slightly later
Draw the difference and the choice is obvious. The edge-triggered controller has exactly as many chances to be correct as there are arrows — miss one and nothing later repairs it. The level-triggered one has no single critical moment, so a lost, duplicated or out-of-order message costs it nothing.
Why this matters for everything below

Because missed and duplicate messages are harmless, the machinery in the rest of this page is allowed to take shortcuts that would otherwise be dangerous: it can throw away the contents of a notification, merge a hundred notifications into one, handle them out of order, and restart from nothing after a crash. Each of those is a real optimisation that only works because of this one decision.

Section 03

The whole machine, at a glance

Four parts, in a line. Read the picture left to right, then come back for the detail sections.

the cluster the API server 1 changes 2 · INFORMER watches everything you care about keeps a local copy up to date and announces each change the local copy (cache) read through a Lister — step 5 name only 3 · WORKQUEUE a to-do list of names to re-check duplicates merge automatically 4 · WORKERS take one name at a time one worker per name 5 · read current state from the cache, not the cluster your Reconcile function compare desired vs actual, then act 6 · writes go straight to the cluster — never through the cache the write in step 6 causes a new change in step 1, which is how the loop keeps turning notice: between step 2 and step 3 the object itself is dropped — only its name travels onward
The one detail worth noticing on a first read. Between the informer and the workqueue, the changed object is thrown away and only its name continues. Everything surprising about controllers follows from that — including why your Reconcile function is handed a name and has to look the object up itself.

Section 04

Informers — the part that watches

An informer is the component that keeps your controller in touch with the cluster. Asking the cluster a question every time you need to know something would be far too expensive, so the informer does something smarter: it asks for everything once, then subscribes to a live feed of changes, and maintains its own copy locally.

Inside, it is four smaller pieces in a line.

REFLECTOR the only part that talks to the cluster 1. fetch everything once 2. subscribe to changes QUEUE OF CHANGES a short ordered list of what just happened added / updated / deleted internal — you never see this INDEXER · THE CACHE an in-memory copy of everything you watch updated first — before step 4 this is what a Lister reads EVENT HANDLERS announce the change to your code your handler writes down the name and stops the cache is always updated before anyone is told that ordering is what makes it safe for the handler to forget the object and pass on only a name
Four pieces, one direction. The dashed arc is the guarantee that ties the design together: by the time your code is told something changed, the local copy already contains the change. So your code never needs the notification's contents — it can always just look.
Worth knowing early

Informers are shared. If five controllers in the same process all watch Pods, they share one informer, one subscription and one copy in memory — not five. This is why adding another controller to an existing operator is usually much cheaper than it sounds.

Section 05

Listers — reading without asking the cluster

A Lister is simply the read interface to that local copy. When your code asks for an object or a list of objects, the answer comes out of memory. No network request is made, nobody waits, and the cluster never learns the read happened.

If you use controller-runtime you may never type the word Lister — the client the framework hands you does this for you — but it is the same mechanism underneath.

your code asks: “give me that object” the local copy answers from memory no request is made the cluster never hears about this read the informer’s one subscription keeps the copy fresh — the only ongoing connection Measured in chapter 03: 2000 reads through a Lister produced 0 requests to the cluster. The same 2000 reads sent directly produced 2000. An informer costs 2 requests for its whole lifetime.
This is why clusters survive having hundreds of controllers. Almost every read a controller performs ends in its own memory. The cost of watching something is paid once, at startup, and never again.
The price of the trick

The local copy is always slightly behind the cluster — changes have to travel back over the subscription before it knows. So immediately after your controller writes something, reading it back may still show the old value. This is normal and not an error. Because the controller is level-triggered, it simply gets nudged again shortly afterwards and sees the new value then.

Section 06

The workqueue — a to-do list of names

When the informer announces a change, your event handler does something that surprises most people the first time they see it: it writes down only the object's name and namespace, puts that on a queue, and throws the object itself away.

That sounds wasteful. It is the single most important design decision in the whole machine, because a queue of names behaves in ways a queue of objects cannot.

five changes to the same object, moments apart replicas 3→4 label added replicas 4→5 status updated replicas 5→3 each becomes the same name THE QUEUE HOLDS default/my-app ×1 five notifications, one item of work if the queue held the objects instead ×5 no two are identical, so nothing can merge — and the order they are applied in starts to matter names are interchangeable; objects are not. That is the whole reason the queue holds names.
Five notifications, one piece of work. Because every entry for an object is the identical name, the queue can recognise duplicates and merge them. Under load this is what stops a busy object from generating unbounded work — and it only works because a name carries no information that could be lost by merging.
What the queue also does

Alongside merging duplicates, the queue makes sure no two workers ever handle the same name at the same time, and it can hold a failed item back for a while before offering it again, waiting longer after each successive failure. All three behaviours are covered in chapters 04 and 05.

Section 07

Workers — the part that does the work

A worker is a loop. It takes one name off the queue, looks up that object's current state in the local copy, works out what needs to change, and does it. Then it reports how that went and takes the next name. Several workers run side by side, so different objects are handled in parallel — but any single object is only ever handled by one worker at a time.

The function you write as an operator author — Reconcile — is the middle step of this loop. Everything around it is provided.

take one name off the queue look it up in the local copy Reconcile compare desired vs actual, then act — your code worked done — wait for next nudge failed try again, after a delay unfixable stop retrying back onto the queue — each failure waits longer than the last several workers run at once, so different objects progress in parallel — but never two workers on the same name note there is no “finished” state: succeeding just means nothing is owed until the next change arrives
Three ways out, and only one of them ends the story. Success means the object is left alone until something changes it again. Failure means it comes back, and waits longer each time. The third exit exists for errors that retrying can never fix — a malformed request, for instance — so that one broken object does not retry forever.

Section 08

What each part actually buys you

The clearest way to understand a part is to ask what would go wrong without it.

PartIn one sentenceWithout it
Informer Watches the cluster and keeps a live local copy of what you care about. You would poll the cluster on a timer — slow to notice changes, and expensive for everyone.
Lister Reads that local copy, so lookups cost nothing and never leave the process. Every read becomes a network request; a few hundred controllers would overwhelm the cluster.
Workqueue A to-do list of names that merges duplicates and hands each name to one worker at a time. Repeated changes to one object would pile up as separate work, and two workers could fight over it.
Workers The loop that takes a name, reads current state, and closes the gap — then retries sensibly on failure. You would hand-roll concurrency, retries and backoff in every controller you write.
The thread running through all four

Every one of these is only safe because the controller is level-triggered. Keeping a slightly-stale local copy, merging five notifications into one, handling objects in an unpredictable order, and starting again from nothing after a restart would all be unacceptable risks in a system that acted on individual changes. In a system that always re-reads the current state, they are free.

Section 09

Glossary

TermWhat it means here
desired stateWhat you asked for, stored in the cluster.
actual stateWhat is really running right now.
reconcileOne pass of comparing the two and closing the gap.
level-triggeredActing on the current value, not on the change. What Kubernetes does.
edge-triggeredActing on the change itself. What Kubernetes deliberately avoids.
informerThe component that watches the cluster and maintains a local copy.
reflectorThe part of an informer that actually holds the connection to the cluster.
indexer / cacheThe in-memory copy of the objects you watch.
listerThe read-only way to query that copy.
event handlerYour callback, run when something changes. Usually it just records a name.
workqueueThe to-do list of names waiting to be reconciled.
workerA loop that pulls names off the queue and reconciles them.
resyncA periodic re-announcement of everything already in the local copy. Not a refresh from the cluster — see chapter 03.
backoffWaiting longer before each successive retry of something that keeps failing.

Section 10

Where to go next

This set

Each chapter takes one part apart and measures it, so the claims on this page can be checked rather than believed.

ch. 01Level, Not Edgehow much work the level-triggered design actually saves, and when it saves none ch. 02The Queue That Changedinside the informer, and what changed there recently ch. 03The Cache You Read Fromwhat reads really cost, and why resync is not a safety net ch. 04Dirty and Processinghow the queue merges duplicates and keeps workers off each other ch. 05Forget Is Not Deletethe worker loop, retries, and the mistake that silently kills an object

Good places to start outside this set

All links checked on 2026-09-11 unless noted.

slidesEdge vs. Level triggered logicTim Hockin · the clearest short explanation of section 02, from a Kubernetes co-founder figureThe official controller diagramthe upstream picture of section 03 — denser than this page, worth revisiting later bookKubebuilder Book · Architecturethe same parts, named as the framework names them primaryWriting Controllersthe official guideline list — short, and every line is earned videoKubernetes Controllers | Informers, Queues and CallbacksJMMTechnology · a walk through the same four parts articleHow the controller-runtime Cache Actually Workskubernetes.io · 2026-07-29 · the next level of detail on sections 04–05 bookProgramming KubernetesHausenblas & Schimanski, O’Reilly · the standard book-length treatment (publisher blocks automated checks)

Closing

What this page simplifies

An orientation page earns its clarity by leaving things out. These are the ones worth knowing about before you rely on the picture above:

  • The informer’s internal queue is more subtle than “a list of changes”, and its behaviour changed meaningfully in recent Kubernetes versions — including one case where a deletion could be announced to nobody. Chapter 02.
  • “The cache is slightly behind” hides a real failure mode. After a restart it is empty rather than stale, and a controller that acts on an empty cache can conclude things were deleted. Chapter 03.
  • Merging duplicates is not always active. When a controller is keeping up, every notification gets its own reconcile and nothing merges at all. Chapter 01.
  • “Retry after a delay” has a ceiling, and reaching it means a broken object is retried only every 16 minutes 40 seconds. Chapter 05.
  • This page describes one controller. Real operators run several, sharing informers and caches, sometimes watching each other’s objects. The shape stays the same; the bookkeeping does not.

Check yourself

If you can answer these without looking, this page has done its job.

Why does the event handler throw away the changed object and keep only its name?

A good answer mentions

  • Names are identical for repeated changes, so the queue can merge them into one piece of work
  • The object can always be looked up again from the local copy, cheaply
  • It is safe because the controller re-reads current state anyway

check against §03 and §06

Your controller misses a notification entirely. How much trouble are you in?

A good answer mentions

  • Very little — it reacts to current state, not to individual notifications
  • The next nudge for that object causes a full re-read, which repairs everything
  • An edge-triggered design would be permanently wrong instead

check against §02

You create something, immediately read it back, and it is not there. What happened?

A good answer mentions

  • Writes go to the cluster; reads come from the local copy
  • The change has to travel back over the subscription before the copy knows
  • Nothing failed — and the controller will be nudged again shortly

check against §05