The Kubelet Checkpoint API  ·  Chapter 00  ·  start here

Three Layers of State

A backup captures what you declared. A volume snapshot captures what you stored. A checkpoint captures what your process was doing — the bytes in its heap, the file descriptors it held, the point in the code it had reached. These are three different layers, three different tools, and they do not substitute for one another. This page defines every term the rest of the set uses, in pictures, with no code.

orientation chapter — no new measurements
every number here is carried from a measured run in chapters 01–05, which show the workings
concepts as of Kubernetes v1.36 · CRI-O 1.36.5 · CRIU 4.2.1

Section 01

Three layers of state, not one

When people say "back up the cluster" they usually mean one specific layer and assume it covers the others. It does not. Separating them is the whole point of this page.

DECLARED STATE Deployments, Services, ConfigMaps, Secrets lives in the API server / etcd · recreatable from YAML VOLUME STATE files a container wrote to a PersistentVolume lives on a disk somewhere · survives the pod RUNTIME STATE heap, stack, open file descriptors, sockets, child processes, the current instruction exists only in RAM · normally destroyed when the pod stops backup tools e.g. Velero · API objects volume snapshots CSI · block/filesystem checkpoint CRIU · this set
Each row has its own tool, and none of them reaches down a row. Restoring every API object gives you a pod that starts from the beginning. Restoring a volume gives you the files but not the process that had them open. Only the bottom row captures what the program was in the middle of doing.

Section 02

What each tool actually captures

ToolCapturesDoes not captureRestores to
Backup (API objects)Specs, labels, Secrets, the declared intentAnything in RAM; anything written since the backupA cluster that will start the workload fresh
Volume snapshotThe bytes on a PersistentVolume at an instantProcess memory; open descriptors; in-flight writes still buffered in the appA disk you can attach to a new pod
Checkpoint (CRIU)Process memory, descriptors, child processes, the exact execution pointAnything the kernel cannot re-create — see chapter 04A container that continues rather than starts
The distinction that matters operationally

A backup answers "what did we ask for?" A checkpoint answers "what was it doing?" If a pod is misbehaving at 03:00 and you want to know why, a backup of its Deployment tells you nothing you did not already have in git. A checkpoint tells you what was in its memory at that moment.

Section 03

What a checkpoint actually is

Concretely: the kubelet exposes an endpoint. You POST to it naming a namespace, a pod and a container. The kubelet asks the container runtime to checkpoint that container; the runtime asks CRIU to dump the process; CRIU walks the process's memory and kernel state and writes it out; the runtime packs the result, plus a filesystem diff and the container's configuration, into a single tar file on the node.

you an HTTP POST kubelet checks permission CRI container runtime CRI-O CRIU dumps the process one .tar file on the node’s disk the container keeps running measured: Running, 0 restarts, counter kept advancing the process is frozen only for as long as the dump takes — measured at 97 ms for a small container (chapter 03)
A checkpoint is non-destructive. The container is frozen, copied, and thawed. Chapter 01 shows the request and the archive; chapter 02 shows why only one runtime can do this today.

Section 04

The asymmetry: you can save, but there is no load button

This is the single most surprising thing about the feature, and the reason the set exists. Kubernetes gives you an API to create a checkpoint. It gives you no API to restore one. The way back is not a Kubernetes verb at all: you wrap the archive in a container image and start a pod from it, which is a container-runtime capability.

running pod one API call POST /checkpoint/… archive on node no API call exists for this direction build an OCI image you do this yourself pod referencing it ordinary pod spec container continues measured: counter 1158 → 1182 one direction is a Kubernetes feature; the other is something you assemble
Save is a verb; load is a procedure. The lower path is not a fallback — it is the only documented way back, and it runs entirely in userland. Chapter 03 measures it end to end and shows a restored process resuming its counter mid-count.

Section 05

Why the feature is called "forensic"

The upstream enhancement is titled Forensic Container Checkpointing, and the word is doing real work. The design target is: a container is behaving suspiciously, you want a copy of its memory to analyse elsewhere, and you want it without alerting whatever is running inside — so the container is not stopped.

That framing explains two things that otherwise look like omissions. There is no restore verb because restoring was never the goal. And the archive is readable by anyone who can read it: it is meant to be carried to an analysis machine.

Carry this into chapter 05

If a checkpoint contains process memory, then it contains everything the process had in memory — including credentials it read from a Secret, decrypted data, and session tokens. Chapter 05 puts a known value into a pod through a Kubernetes Secret and recovers it from the archive with one command. Treat a checkpoint file as being as sensitive as a memory dump, because that is what it is.

Section 06

Glossary

TermMeaning in this set
declared stateAPI objects — what you asked the cluster for.
runtime stateWhat exists only inside a running process: memory, descriptors, sockets, children.
checkpointA capture of runtime state, written to a tar archive on the node.
restoreStarting a container from a checkpoint so the process continues rather than starts.
CRIUCheckpoint/Restore In Userspace — the Linux tool that does the actual dumping.
CRIContainer Runtime Interface — the gRPC contract between the kubelet and the runtime.
CRI-OA container runtime. As of this writing, the only one implementing the checkpoint CRI call (chapter 02).
dumpCRIU's word for capturing a process. The opposite of restore.
rootfs diffThe files the container changed since its image — shipped inside the archive.
feature gateA kubelet switch. ContainerCheckpoint has been on by default since 1.30.
KEPKubernetes Enhancement Proposal. This feature is KEP-2008.

Section 07

Where to go next

This set

Chapters 01–05 were measured on a real single-node cluster; chapter 06 is documentary.

ch. 01One POST, One Tarthe endpoint, the permission it needs, and what is inside the archive ch. 02Only One Runtimekubelet → CRI → runtime → CRIU, and the three places it refuses ch. 03The Missing Verbrestore as a userland procedure; size and time measured both ways ch. 04What Comes Backthe survival table — and the state that blocks the checkpoint entirely ch. 05The Archive Is Memoryrecovering a Secret from a checkpoint with one command ch. 06Where It Is GoingKEP status, the working group, and how live migration differs

Primary sources worth reading first

articleForensic Container Checkpointing in Kuberneteskubernetes.io · 2022-12-05 · the original announcement (alpha-era; see ch. 01 for what changed) articleForensic container analysiskubernetes.io · 2023-03-10 · what to do with an archive once you have one articleAnnouncing the Checkpoint/Restore Working Groupkubernetes.io · 2026-01-21 · where the feature is heading primaryCRIU — What cannot be checkpointedthe authoritative list behind chapter 04 KEPKEP-2008the enhancement proposal itself primaryCRIU project documentationthe tool that does the work

Closing

What this page simplifies

  • "The runtime asks CRIU" hides three separate places the request can be refused — a feature gate, a runtime that does not implement the call, and a runtime binary built without CRIU support. Chapter 02.
  • "Non-destructive" is a property of this code path, not of checkpointing in general. CRIU can absolutely stop a process; the Kubernetes path hardcodes keeping it running. Chapter 01.
  • Not all runtime state survives. One common kind does not merely fail to come back — it stops the checkpoint being taken at all. Chapter 04.
  • "Wrap it in an image and start a pod" is three commands and at least one configuration change that is not obvious. Chapter 03.

Check yourself

Your cluster has nightly Velero backups and hourly CSI volume snapshots. A pod is leaking memory and you want to know what is in it. Which tool helps?

A good answer mentions

  • Neither — both capture layers above the process
  • Velero gives you the spec you already have in git; the snapshot gives you files, not heap
  • A checkpoint is the only one that captures process memory

check against §01 and §02

Why is there an API to take a checkpoint but not to restore one?

A good answer mentions

  • The design goal was forensic analysis, not resumption
  • Restoring is a container-runtime capability reached by building an image
  • So the return path is userland, and is not a Kubernetes verb

check against §04 and §05