Dirty and Processing

Section 01

Three sets, and the invariant that ties them

The workqueue is not a list with a deduplication check bolted on. It is three structures maintained together, and the comment on the first one states the invariant the other two exist to preserve.

// queue defines the order in which we will work on items. Every
// element of queue should be in the dirty set and not in the
// processing set.
queue Queue[t]

// dirty defines all of the items that need to be processed.
dirty sets.Set[t]

// Things that are currently being processed are in the processing set.
// These things may be simultaneously in the dirty set. When we finish
// processing something and remove it from this set, we'll check if
// it's in the dirty set, and if so, add it to the queue.
processing sets.Set[t]

dirty answers “does this need work?”. processing answers “is someone on it?”. queue is only the ordering, and it is allowed to hold an item only when the first answer is yes and the second is no. Every behaviour in this chapter falls out of maintaining that.

Section 02

One hundred thousand adds, one entry

Deduplication is a single set lookup at the top of Add. It costs one hash probe and it is unconditional.

func (q *Typed[T]) Add(item T) {
	...
	if q.dirty.Has(item) {
		// the same item is added again before it is processed, call the Touch
		// function if the queue cares about it (for e.g, reset its priority)
		if !q.processing.Has(item) {
			q.queue.Touch(item)
		}
		return
	}

	q.metrics.add(item)

	q.dirty.Insert(item)
	if q.processing.Has(item) {
		return
	}

	q.queue.Push(item)
	q.cond.Signal()
}

A. Add("default/target") x 100000
   queue Len() = 1

Section 03

Where adds go while a worker holds the key

The interesting case is not a quiet queue. It is 200 events arriving for an object that is being reconciled right now — which is the normal condition under load, as chapter 01 measured.

B. one worker holds the key, then 200 more Add()s arrive
   Len() while key is held, after 200 adds : 0   <- adds went to `dirty`, not the queue
   Len() observed by the worker before Done: 0
   Len() immediately after Done()          : 1   <- Done() re-queued it, once
   further Get()s needed to drain          : 1   <- 200 adds produced exactly this many reconciles

func (q *Typed[T]) Done(item T) {
	...
	q.processing.Delete(item)
	if q.dirty.Has(item) {
		q.queue.Push(item)
		q.cond.Signal()
	} else if q.processing.Len() == 0 {
		q.cond.Signal()
	}
}

Section 04

Serial per key, parallel across keys

The controller-writing guide promises that “no two goroutines will work on the same item at the same time.” That is a concurrency claim, so it deserves a concurrency test: sixteen workers, sixty-four keys, eight producers, and a detector that flags any instant two workers hold the same key.

queue implementation            : client-go workqueue
workers                         : 16
distinct keys                   : 64
Add() calls from 8 producers    : 320000
reconciles actually executed    : 148497
collapse ratio                  : 2.2x
max distinct keys in flight     : 14  (workers=16)
SAME-KEY CONCURRENCY VIOLATIONS : 0   <- must be 0
elapsed                         : 1.793s

Section 05

The control that fails on purpose

A detector that reports zero is worthless until you have seen it report something. The same harness, with the workqueue swapped for a plain buffered channel — no dirty, no processing, no Done — is the control.

queue implementation            : plain channel (CONTROL, expected to fail)
workers                         : 16
distinct keys                   : 64
Add() calls from 8 producers    : 320000
reconciles actually executed    : 320000
collapse ratio                  : 1.0x
max distinct keys in flight     : 16  (workers=16)
SAME-KEY CONCURRENCY VIOLATIONS : 21940   <- must be 0
elapsed                         : 576ms

Section 06

Further reading

All links returned HTTP 200 on 2026-09-11 unless noted.

Section 07

Closing note — what varies, and what was not verified

Durable versus run-specific

• Durable: Len() == 1 after 100,000 adds; Len() == 0 while the key is held regardless of how many adds arrive; Len() == 1 immediately after Done; exactly one further reconcile; 0 same-key violations with the workqueue and a non-zero count with the channel control.
• Run-specific: 148,497 reconciles, the 2.2× collapse ratio, 14 keys in flight, 21,940 violations, and both elapsed times. All depend on scheduler timing, core count, and the 20 µs artificial hold. The signs are durable: workqueue collapse > 1× with 0 violations; channel collapse = 1.0× with many.

Traps this chapter had to avoid

• A detector that cannot fail. Reporting “0 violations” from a harness never observed failing proves nothing — it is indistinguishable from a broken detector. The -naive control exists solely to make the zero meaningful, and it is a demo designed to fail.
• Holding the key too briefly to overlap. With no artificial hold, workers finish before another can collide, and even the broken control reports zero. The 20 µs sleep exists to make overlap possible; without it the test is vacuous in both arms.
• Reading Len() as “outstanding work”. It is the queue slice only. During part B it reads 0 while 200 changes are pending.

Not verified

• The priority queue. §02–§05 measure the plain client-go Typed workqueue. controller-runtime defaults to its own priority queue (UsePriorityQueue, default true), which wraps the same dirty/processing discipline but adds ordering by priority. Chapter 05 covers it; nothing here measured it.
• Rate limiting and delayed adds. AddRateLimited, AddAfter and the backoff curve are chapter 05's subject and were not exercised here.
• Metrics behaviour. q.metrics.add / done appear in the quoted source; no metrics provider was registered, so their behaviour was not observed.
• Shutdown semantics. ShutDown versus ShutDownWithDrain differ in whether in-flight items are awaited; only the former was used, and not tested for drain behaviour.

Section 08

Spoken drills