The Kubelet Checkpoint API  ·  Chapter 04  ·  measured, not recalled

What Comes Back

An in-memory counter, an open file descriptor and a child process all survived a checkpoint and restore into a different pod — the child even kept the same PID. An established TCP connection did not merely fail to come back: it stopped the checkpoint being taken at all, with CRIU refusing and suggesting an option the Kubernetes path has no way to pass. That asymmetry — three kinds of state restored perfectly, one that blocks the operation outright — is the finding of this set.

measured on Kubernetes v1.36.4 · CRI-O 1.36.5 · CRIU 4.2.1 · runc 1.5.1 · kernel 6.8.0-139-generic
each variant run twice; both runs agreed on every outcome
mode A — subject written in Go, one state type per variant; see demos/subject

Section 01

The experiment

One small Go program holds four kinds of state that exist only inside a running process, each switchable independently so that a failure can be attributed to one of them. It prints a status line every second, so the whole history is visible in the container log.

  • Memory — a counter incremented in a local variable, written nowhere.
  • Open file descriptor — a file created, written, and held open.
  • Child process — a forked child, probed for liveness with signal 0.
  • Established TCP — a connection to a peer pod, kept open and written to.

For each variant: run it, let it accumulate state, checkpoint via the kubelet endpoint, wrap the archive as an image, start a new pod from it, and read what the restored process reports.

Section 02

The table

demos/scripts/survive-matrix.sh — run 1 of 2rung 1 · measured
memory       | ok  86ms | 893440 B | rss= 2844 kB | ratio=.30 | RESUMED (tick 11 -> 21, 1 BOOT)
    before : tick=11 child=none fd=none tcp=none
    after  : tick=21 child=none fd=none tcp=none

memory-file  | ok  95ms | 857088 B | rss= 2808 kB | ratio=.29 | RESUMED (tick 11 -> 21, 1 BOOT)
    before : tick=11 child=none fd=ok(3) tcp=none
    after  : tick=21 child=none fd=ok(3) tcp=none

memory-child | ok 104ms | 945664 B | rss= 3080 kB | ratio=.29 | RESUMED (tick 11 -> 21, 1 BOOT)
    before : tick=11 child=alive(12) fd=none tcp=none
    after  : tick=21 child=alive(12) fd=none tcp=none

memory-tcp   | CHECKPOINT REFUSED after 92ms | rss=3260kB
      reason: Error (criu/sk-inet.c:200): inet: Connected TCP socket
    before: tick=11 child=none fd=none tcp=ok(10.244.0.26:49856)
bash demos/scripts/survive-matrix.sh
Runtime stateCheckpointRestoreEvidence
In-memory countersucceedssurvivestick continues 11 → 21, exactly one boot banner
Open file descriptorsucceedssurvivesfd=ok(3) still writable after restore
Child processsucceedssurvivesalive(12) — same PID as before the checkpoint
Established TCP connectionrefused— never reachedcriu/sk-inet.c:200
Finding — the failure mode is the wrong shape

Intuition says a checkpoint captures what it can and drops the rest. That is not what happens. Three state types came back perfectly — not approximately, not partially. The fourth did not degrade: it aborted the whole operation, leaving no archive at all.

So “what does not survive a checkpoint” is the wrong question for this state. The right one is “what prevents a checkpoint”, and a networked service with an open connection — which is most of them — is on that list.

Section 03

What survived, and how completely

ORIGINAL POD RESTORED POD checkpoint → archive → restore counter = 11 counter = 12, 13, … 21 fd 3 open, written fd 3 still writable child PID 12 alive child PID 12 alive same number TCP to peer:9000 established no archive is written the checkpoint call returns an error
Three arrows cross; one stops before the boundary. The child keeping PID 12 is the detail worth pausing on — CRIU restores the process tree with its original identifiers, which is why a parent’s stored PID is still valid after the round trip.
How resumption was judged

Not by “the pod is Running”. CRI-O restores the container log along with the process, so a restored container’s log begins with the original boot banner and looks like a fresh start. The test used here is stricter: exactly one boot banner in the whole log, and a monotonic counter that never resets. An earlier version of this experiment got the answer wrong by reading the first log line.

Section 04

The refusal, in CRIU’s own words

the checkpoint response for the TCP variantrung 1 · measured
sockets: Searching for socket 0x15a79 family 2
Error (criu/sk-inet.c:200): inet: Connected TCP socket, consider using --tcp-established option.
Error (criu/cr-dump.c:1545): Dump files (pid: 19872) failed with -1
Error (criu/cr-dump.c:1975): Dumping FAILED.

criu failed: type DUMP errno 0
HTTP 500

CRIU is not saying this is impossible. It is saying there is a flag for it — --tcp-established — and that it will not proceed without being told explicitly, because freezing one end of a live connection has consequences the caller must opt into.

Section 05

Why you cannot pass that flag

This is where chapter 02’s reading pays off. The suggestion in CRIU’s error is unreachable from Kubernetes, and the reason is structural.

CheckpointContainerRequest container_id location timeout no options field ContainerCheckpointOptions Keep KeepRunning TargetFile no options field CRIU accepts --tcp-established and many other flags capable, but never told the flag CRIU asks for has nowhere to travel — there is no field for it at any hop
The capability exists; the channel does not. CRIU can checkpoint established connections. The Kubernetes path narrows the request to three fields for the forensic use case, and a consequence of that narrowing is that an entire class of workload cannot be checkpointed through the kubelet at all.
Finding — what this rules out

Any container holding an open TCP connection at the moment you call the endpoint cannot be checkpointed. That includes most database clients with a connection pool, most service meshes’ sidecars, anything with a long-poll or watch open, and — awkwardly for a forensic feature — anything mid-exfiltration over a socket.

The practical shape of this: checkpointing is reliable for compute-ish workloads that are momentarily idle on the network, and unreliable-by-construction for connected services. That is a much narrower feature than “capture any container’s memory”.

Section 06

CRIU’s own list of what it will not dump

The TCP case is one entry on a documented list. The rest is worth knowing before planning around this feature.

CategoryExamples
Needs an explicit optionExternal resources; file locks (--file-locks); invisible files; established TCP (--tcp-established)
Cannot be dumpedOpen or mapped character/block devices (virtual ones like null/zero and TUN excepted); open files from a lazily unmounted filesystem; tasks with a debugger attached (ptrace has one holder, so gdb or strace blocks a dump); tasks of another user when CRIU is non-root; sockets other than TCP, UDP, UNIX, packet and netlink; packetized (O_DIRECT) pipes; cork-ed UDP sockets; files passed over UNIX sockets; SysVIPC segments without an IPC namespace

source: criu.org, What cannot be checkpointed, retrieved 2026-09-13  ·  rung 3

The page’s own framing

“Note that there is no ‘What cannot be restored’ article, and never will be. If something was dumped, it should be restored.” That is exactly the asymmetry this chapter measured: the hard edge is at dump time, and once you have an archive, restoration is expected to work.

Section 07

Further reading

primaryCRIU — What cannot be checkpointedthe list in §06, from the project itself primaryCRIU — TCP connectionwhat --tcp-established does and what it requires of the peer primaryCRIU — command line referencethe options the Kubernetes path cannot reach primaryCRIU — sk-inet.cthe file and check that refused the dump in §04 articleForensic container analysiskubernetes.io · what analysis of a successful archive looks like articleCheckpoint/Restore Working Groupkubernetes.io · 2026-01-21 · several listed use cases require exactly the connected workloads this chapter cannot capture

Section 08

Closing note — what varies, and what was not verified

Durable versus run-specific

  • Durable: all four outcomes. Both runs of the matrix agreed: three resumed, TCP refused, same CRIU error line. The child keeping its PID is a property of how CRIU restores process trees, not an accident of this run.
  • Run-specific: archive sizes (843–976 KB), times (86–104 ms), the specific PID and port numbers, and the tick values, which depend only on how long the script waited.

Design limits of this experiment

  • One connection, one child, one descriptor. Nothing here tests many of each, or interactions between them.
  • The peer was inside the cluster. A connection to something outside the cluster was not tried; CRIU’s refusal happens before any of that matters, but a successful --tcp-established dump would care a great deal.
  • Restore was to the same node. The child’s PID was free to be reused there. On a different node with a different PID namespace population, PID collisions are a real restore failure mode and were not tested.
  • The file descriptor pointed at a regular file in the container’s own filesystem. A descriptor on a mounted volume, a pipe, or a UNIX socket was not tested.

Not verified

  • Whether --tcp-established would in fact succeed here. It cannot be passed, so the claim is only that Kubernetes cannot ask — not that CRIU would have managed it on this kernel.
  • UDP, UNIX and netlink sockets. CRIU lists them as dumpable; none were exercised.
  • The other entries in §06. Quoted from CRIU’s documentation (rung 3), not reproduced.

Section 09

Spoken drills

You are asked to checkpoint a production API server pod for analysis. What do you expect to happen?

A strong answer hits

  • Most likely a refusal — it will have established TCP connections
  • CRIU stops at sk-inet.c and suggests --tcp-established
  • No archive is produced at all; this is not partial capture
  • The option cannot be passed: neither the CRI request nor CRI-O’s options struct has a field for it
  • The honest half: if the pod happens to have no established connection at that instant, it will succeed — so this can look intermittent rather than structural, which is worse for whoever debugs it

check against §04 and §05

What actually came back from a successful restore, and how sure are you?

A strong answer hits

  • Memory: a counter continued 11 → 21 rather than resetting
  • Open descriptor: still writable, same number
  • Child process: alive, and with the same PID
  • Confidence comes from one boot banner in the log plus a monotonic counter, not from pod status
  • The honest half: CRI-O replays the original container log, so a restored pod’s log looks like a fresh boot. Reading the first line is how you get this wrong

check against §02 and §03

Why is “what does not survive a checkpoint” a slightly wrong question?

A strong answer hits

  • The hard edge is at dump time, not restore time
  • State that CRIU cannot capture stops the checkpoint rather than degrading it
  • CRIU’s docs say as much: if it was dumped, it should restore
  • The honest half: restore can still fail for environmental reasons — a scratch-based image failed to restore on this stack with a mount-mapping error. So “dumped implies restorable” is the design intent, not a guarantee

check against §02 and §06

Which workloads is this feature actually reliable for?

A strong answer hits

  • Compute-shaped work that is momentarily idle on the network — batch jobs, simulations, notebook kernels between requests
  • Not connected services: connection pools, sidecars, long-polls, watches
  • Also excluded: anything holding a device, anything under a debugger, anything with exotic sockets
  • The honest half: several of the working group’s stated use cases — pod migration, interruption-aware scheduling — need exactly the connected workloads that the current path refuses, which is a fair read of why the work is continuing

check against §05 and §06