The Kubelet Checkpoint API  ·  Chapter 03  ·  measured, not recalled

The Missing Verb

The word restore appears zero times in the kubelet’s HTTP server — not in 1.36, not on master. There is no endpoint, no verb, nothing to call. The way back is a procedure you assemble yourself: wrap the archive in an OCI image and start a pod from it. It does work — a restored container here picked its counter up at 1182 having been checkpointed at 1158 — but it took a CRI-O configuration change to get there, and the archive costs about 1× the container’s resident memory.

measured on Kubernetes v1.36.4 · CRI-O 1.36.5 · CRIU 4.2.1 · runc 1.5.1
Ubuntu 24.04.5 aarch64, kernel 6.8.0-139-generic, 4 vCPU / 6 GiB
mode A — both directions executed; source checked at release-1.36 and master

Section 01

There is no restore verb

The simplest possible check, and the most conclusive one.

kubernetes/kubernetes@master · pkg/kubelet/server/server.gorung 2 · read the source
$ grep -in "restore" pkg/kubelet/server/server.go
(no matches)

$ grep -n "checkpoint" pkg/kubelet/server/server.go
117:	checkpointPath      = "/checkpoint/"
692:	// Only enable checkpoint API if the feature is enabled
1278:func (s *Server) checkpoint(request *restful.Request, response *restful.Response) {

One direction is an endpoint. The other is not represented in the kubelet’s HTTP surface at all. This is not an oversight — it follows from the feature’s stated purpose. KEP-2008 is titled Forensic Container Checkpointing; the goal was to obtain a copy for analysis, not to resume execution.

Section 02

What master adds — and what it does not

The CRI is moving. It is worth being precise about what exists where, because it is easy to read a proto file on main and describe a capability no released cluster has.

cri-api: release-1.36 (under test) versus masterrung 2 · read the source
                        release-1.36   master
CheckpointContainer          1           1
CheckpointPod                0           1
RestorePod                   0           1
grep -c "rpc RestorePod(" api.proto # on each branch

So a RestorePod RPC is arriving — pod-level checkpoint/restore work, with its own proto comment describing a freeze-all / resume-all contract and requiring restored containers to be “in the CREATED state and must not have executed the restored process”. Two cautions:

  • It is on master, not in the release tested here.
  • It is a CRI RPC — an interface between kubelet and runtime. It is not a kubelet HTTP endpoint, and the kubelet server still contains no restore.
Finding — state the version with the claim

“Kubernetes can restore containers” is false on 1.36 as an API statement and misleading even on master. What is true: CRI-O can restore a container, and you reach that capability by constructing an image, not by calling Kubernetes.

Section 03

The procedure that stands in for the verb

Three steps, all userland. The archive is wrapped in a scratch image carrying one annotation, and a pod references that image like any other.

checkpoint tar on the node, 0600 1. wrap in an image buildah from scratch buildah add <tar> / commit to local storage 2. one annotation io.kubernetes.cri-o. annotations.checkpoint.name = the container name 3. an ordinary pod image: that image CRI-O recognises it and restores nothing in this row is a Kubernetes feature. The pod spec is ordinary; the intelligence is entirely in CRI-O, which inspects the image, finds the checkpoint annotation, and takes the restore path instead of the normal start path.
The pod spec is the same shape as any other. Which is the point, and the risk: nothing in the manifest tells a reader that this pod resumes a process rather than starting one. The only marker is an annotation inside the image.

Section 04

The blocker nobody mentions

With the image built correctly, the restored pod would not start:

kubectl describe pod restoredrung 1 · measured
Status: CreateContainerError

Warning  Failed  kubelet  Error: namespaced signature policy /etc/crio/policy.json defined
for pods in namespace default; signature validation is not supported for container restore
cri-o@main · server/container_restore.go:91rung 2 · read the source
// WARNING: This hard-codes an assumption that SignaturePolicyPath set specifically for the
// namespace is never less restrictive than the default system-wide policy, i.e. that if an
// image is successfully pulled, it always conforms to the system-wide policy.
if systemCtx.SignaturePolicyPath != "" {
	return "", fmt.Errorf(
		"namespaced signature policy %s defined for pods in namespace %s; signature validation is not supported for container restore", ...)
Finding

CRI-O refuses to restore whenever a signature policy path resolves for the pod’s namespace — and the CRI-O static bundle installs one at /etc/crio/policy.json by default. So a stock install can take checkpoints but cannot restore them until that is cleared. Setting signature_policy = "" made the restore succeed.

Note what that trade is: restoring a checkpoint image bypasses image signature verification, by design, because the archive is not a signed artifact. In a cluster that enforces image signing, this path is a hole worth knowing about before you open it.

Section 05

It really does resume

The test that settles whether this is a restore or a restart: a container incrementing a counter, checkpointed mid-count.

busybox container writing an incrementing counter, checkpointed then restored into a new podrung 1 · measured
counter value before checkpoint : 1158
archive                         : checkpoint-counter_default-app-2026-09-13T17:57:46Z.tar

restored pod status             : Running
counter value in restored pod   : 1182      (~24 s later — it continued)

A restarted container would have begun at zero. The counter carried on from where it was, which means the process’s memory came back. Chapter 04 pushes this much harder — child processes, descriptors, sockets — and finds where it stops.

A false reading worth repeating

An early version of this test concluded “the process restarted” because the restored container’s log began with the original boot banner. It does — CRI-O restores the container log along with the process, so the archive’s log is replayed before execution continues. The reliable signal is that there is exactly one boot banner and the counter never resets, not that the log looks like a fresh start.

Section 06

What it costs

demos/scripts/size-vs-rss.sh — allocating and touching anonymous memoryrung 1 · measured
   alloc       rss_kB      archive_B    ratio         ms
      0M         2908         956416      .32         97
     16M        19528       17915904      .89        119
     64M        68776       68353536      .97        180
    256M       265552      269883392      .99        554
0.25×0.50×0.75×1.00× alloc 0MiB: rss 2908kB, archive 956416B, ratio 0.32, 97msalloc 16MiB: rss 19528kB, archive 17915904B, ratio 0.89, 119msalloc 64MiB: rss 68776kB, archive 68353536B, ratio 0.97, 180msalloc 256MiB: rss 265552kB, archive 269883392B, ratio 0.99, 554ms0.32×0.89×0.97×0.99×0 MiB16 MiB64 MiB256 MiB97 ms119 ms180 ms554 ms archive / RSS anonymous memory allocated and touched by the container (ordinal spacing) amber row: wall-clock time of the checkpoint call converges on 1.0× small process: much of RSS is shared file-backed pages CRIU references rather than copies
The archive is the anonymous memory. Each point is a measured run; hover for the raw figures. The amber row under the axis is the wall-clock time of the checkpoint call, which is also the length of the freeze.
Finding — budget for 1× RSS

For a memory-heavy container the archive is essentially the size of its resident memory. The 0.32× at the small end is not a compression win — it is that most of a tiny process’s RSS is shared, file-backed pages (the binary and libraries) which CRIU references rather than copies. Anonymous memory is copied in full.

Time scales with it too: roughly 2 ms per 10 MiB of anonymous memory on this hardware, on top of a ~90 ms floor. A container with 8 GiB resident should be expected to produce an 8 GiB file and to be frozen for the duration — plan node disk accordingly, because the archive lands on the node.

Section 07

Further reading

articleForensic container analysiskubernetes.io · 2023-03-10 · what to do with an archive, including the image-wrapping recipe primaryCRI-O — container_restore.gothe signature-policy refusal in §04 toolcheckpointctlinspects checkpoint archives without extracting them by hand primarycri-api — api.proto on masterwhere CheckpointPod and RestorePod are appearing (§02) primaryCRIU — Checkpoint/Restorewhat resumption means at the process level toolbuildahused for step 1 of §03

Section 08

Closing note — what varies, and what was not verified

Durable versus run-specific

  • Durable: no restore in the kubelet server; the 1.36-versus-master RPC difference; the signature-policy refusal and its source line; that the counter continues rather than resets; that the ratio approaches 1.0× as anonymous memory grows.
  • Run-specific: 1158 and 1182; every millisecond figure; the exact byte counts. Timings are from a QEMU guest on Apple silicon and will differ substantially on server hardware.

A failure worth recording

A workload image built FROM scratch failed to restore on this stack, with CRIU reporting mnt: No mapping for 1445:(null) mountpoint and the container then starting fresh. The same experiment on a busybox-based image restored correctly. The restore path appears sensitive to the shape of the container’s root filesystem. That was not root-caused here — it is reported as observed, and it is why every measurement in this chapter uses a conventional base image.

Not verified

  • Restore onto a different node. Everything here is single-node. Cross-node restore requires moving the archive and having a compatible kernel and runtime at the far end; not tested.
  • CheckpointPod / RestorePod. Not in 1.36; nothing about them was executed.
  • Restore with volumes. The subject had no PersistentVolume. Interaction between a restored process and reattached storage was not examined.
  • Signature enforcement. It was switched off to proceed; no test was made of restoring under a policy that permits the image.

Section 09

Spoken drills

Does Kubernetes support restoring a checkpointed container?

A strong answer hits

  • Not as an API — the word restore does not appear in the kubelet HTTP server on any version checked
  • The path back is userland: wrap the archive in an OCI image, run a pod from it
  • CRI-O does the restoring; Kubernetes just starts a pod
  • A RestorePod CRI RPC is appearing on master, but that is runtime-facing and not in 1.36
  • The honest half: it genuinely works — the process resumes mid-execution. The absence is of a verb, not of the capability

check against §01, §02 and §05

How much disk and how much pause should you budget for checkpointing a 4 GiB service?

A strong answer hits

  • Roughly 1× resident memory — the ratio measured 0.99 at 256 MiB and is still climbing
  • The archive lands on the node, so it is node disk, not cluster storage
  • Pause scales with memory: ~2 ms per 10 MiB here, plus a ~90 ms floor
  • The honest half: the ratio is low for small processes because shared file-backed pages are referenced, not copied — so you cannot extrapolate down from a big number or up from a small one

check against §06

Your restore fails with CreateContainerError mentioning signature policy. What is happening, and what are you accepting by fixing it?

A strong answer hits

  • CRI-O refuses restore when a namespaced signature policy path resolves
  • The static bundle installs one by default, so a stock node hits this
  • Clearing it lets the restore proceed
  • The honest half: you are accepting that checkpoint images bypass signature verification. In a cluster that enforces signing, this is a real gap — the archive is not a signed artifact and cannot be made into one this way

check against §04

A restored pod’s log starts with the application’s boot banner. Did the restore fail?

A strong answer hits

  • Not necessarily — CRI-O restores the container log too, so the pre-checkpoint log is replayed
  • The signal is whether there is one boot banner or two, and whether counters reset
  • Measured: one banner, counter continuing 1158 to 1182
  • The honest half: this genuinely produced a wrong conclusion during this work. If you are testing restore, instrument something monotonic in memory rather than reading the log’s first line

check against §05