Forget Is Not Delete

Section 01

The loop as it is written today

The shape most tutorials teach is: Get, reconcile, then Forget+Done on success or AddRateLimited+Done on failure. The semantics are right. The spelling in current controller-runtime differs in three ways that matter.

func (c *Controller[request]) processNextWorkItem(ctx context.Context) bool {
	obj, priority, shutdown := c.Queue.GetWithPriority()
	if shutdown {
		return false
	}

	// We call Done here so the workqueue knows we have finished
	// processing this item. ...
	defer c.Queue.Done(obj)

	ctrlmetrics.ActiveWorkers.WithLabelValues(c.Name).Add(1)
	defer ctrlmetrics.ActiveWorkers.WithLabelValues(c.Name).Add(-1)

	c.reconcileHandler(ctx, obj, priority)
	return true
}

• Done is deferred, registered before any reconciling happens. It runs on success, on error, and on panic. It is not a branch of the switch.
• The queue is a priority queue — GetWithPriority, not Get. UsePriorityQueue defaults to true.
• There is a TerminalError path that suppresses the retry entirely, which the classic shape has no equivalent for.

Section 02

The four-way switch

Everything the controller decides after your Reconcile returns happens in one switch. The asymmetry worth noticing is which arms call Forget.

TerminalError is detected with errors.Is, which means it survives wrapping — a practical detail worth confirming rather than assuming.

plain error                      errors.Is(err, TerminalError(nil)) = false   requeued? true
TerminalError                    errors.Is(err, TerminalError(nil)) = true    requeued? false
TerminalError wrapped with %w    errors.Is(err, TerminalError(nil)) = true    requeued? false
nil                              errors.Is(err, TerminalError(nil)) = false   requeued? false

TerminalError message : terminal error: spec.replicas must be >= 0
Unwrap()              : spec.replicas must be >= 0

Section 03

The retry curve a failing object actually experiences

With the priority queue enabled — the default — the rate limiter is NewTypedItemExponentialFailureRateLimiter(5ms, 1000s). Reading the delays straight off it:

failure             delay       cumulative
1                     5ms               0s
2                    10ms               0s
4                    40ms               0s
8                   640ms               1s
10                  2.56s               5s
12                 10.24s              20s
14                 40.96s            1m22s
16               2m43.84s            5m28s
18              10m55.36s           21m51s
19                 16m40s           38m31s
20                 16m40s           55m11s
22                 16m40s         1h28m31s

first failure hitting the 1000s cap: #19

Section 04

What Forget actually touches

The name is the problem. Forget sounds like removal. The doc comment is explicit that it is not.

// Forget indicates that an item is finished being retried.  Doesn't matter whether it's for perm failing
// or for success, we'll stop the rate limiter from tracking it.  This only clears the `rateLimiter`, you
// still have to call `Done` on the queue.
Forget(item T)

1. Forget() does not remove anything from the queue
   after Add("a")            Len()=1
   after Forget("a")         Len()=1   <- still queued
   after Get()+Done()        Len()=0

Section 05

The silent stall

The failure mode follows directly. Get puts the key in processing. Only Done takes it out. Miss it, and every future event for that object lands in dirty and stays there — because, as chapter 04 measured, an add for a key that is currently processing is forbidden from touching the queue.

2. CORRECT: Get -> Forget -> Done. The key can be processed again.
   re-added and retrieved: "obj"  <- healthy

3. BUG: Get -> Forget, but Done() is never called.
   100 further Add("obj") calls, then Len()=0
   Get() timed out after 500ms: the key is NEVER returned again
   => this object has silently stopped reconciling. No error. No log.

Section 06

Further reading

All links returned HTTP 200 on 2026-09-11 unless noted.

Section 07

Closing note — what varies, and what was not verified

Durable versus run-specific

• Durable: the backoff doubling from 5 ms; the cap first reached at failure #19; Forget resetting the delay to 5 ms; Len() unchanged by Forget; the stall in part 3. All are deterministic arithmetic or set operations with no timing dependence.
• Run-specific: only the 500 ms timeout used to demonstrate the stall — the stall itself is permanent, and the timeout is just how long the demo is willing to wait before declaring it.

Traps this chapter had to avoid

• A demo that hangs instead of reporting. Part 3 deliberately creates a queue from which Get never returns. Calling Get directly would hang the program forever and look like a broken demo rather than a finding. It is wrapped in a goroutine with a timeout so the stall is reported.
• Reading the wrong rate limiter. controller-runtime picks a different default depending on UsePriorityQueue. Measuring DefaultTypedControllerRateLimiter and presenting it as “the default” would be wrong for a stock controller, which uses the plain exponential limiter. Both are measured in §03.
• Assuming TerminalError breaks under wrapping. It is detected with errors.Is, so %w is safe — worth testing rather than guessing, since the opposite would be a common and silent bug.

Not verified

• The priority queue's ordering behaviour. GetWithPriority, AddWithOpts and the priority semantics were read in source but not exercised; §03 measures the rate limiter controller-runtime pairs with it, not the queue itself.
• End-to-end through a manager. No Manager, Controller or envtest was started. The switch in §02 is quoted from source (rung 2); only the TerminalError predicate inside it was executed.
• Metrics. ReconcileTotal, ReconcileErrors, TerminalReconcileErrors and ActiveWorkers appear in the quoted code; none were registered or observed.
• Panic behaviour. The claim that deferred Done also runs on panic follows from Go semantics and RecoverPanic exists as an option, but no panicking reconciler was run here.

Section 08

Spoken drills