Only One Runtime

Section 01

Four components, four ways to be told no

A checkpoint request passes through the kubelet, the CRI, the container runtime, the OCI runtime and finally CRIU. Each link can refuse, and each refuses differently.

Each refusal has a different shape. A missing gate gives you no route at all. containerd answers the call and declines it. CRI-O declines before reaching the runtime. The runtime declines before reaching CRIU. Knowing which error belongs to which link is most of the debugging.

Section 02

What actually crosses the CRI

The entire contract between kubelet and runtime is three fields.

k8s@release-1.36 · staging/src/k8s.io/cri-api/.../v1/api.protorung 2 · read the source

message CheckpointContainerRequest {
    string container_id = 1;   // ID of the container to be checkpointed.
    string location     = 2;   // Location of the checkpoint archive used for export
    int64  timeout      = 3;   // Timeout in seconds; 0 = CRI default
}

And CRI-O’s internal options object is three fields as well:

cri-o@main · internal/lib/checkpoint.go:24rung 2 · read the source

type ContainerCheckpointOptions struct {
	// Keep tells the API to not delete checkpoint artifacts
	Keep bool
	// KeepRunning tells the API to keep the container running
	// after writing the checkpoint to disk
	KeepRunning bool
	// TargetFile tells the API to read (or write) the checkpoint image
	// from (or to) the filename set in TargetFile
	TargetFile string
}

Finding — no CRIU option can be passed

There is no field, anywhere in this chain, for CRIU flags. Not in the CRI message, not in CRI-O’s options. That is why chapter 04’s TCP result is a hard limit rather than a tuning problem: CRIU asks for --tcp-established, and the Kubernetes path has no way to give it one.

Section 03

containerd answers the call and declines it

This is the fact that determines whether you can try this at all, and it is worth reading in full because the whole file is short.

containerd@main · internal/cri/server/container_checkpoint.go — the entire implementationrung 2 · read the source

func (c *criService) CheckpointContainer(ctx context.Context, r *runtime.CheckpointContainerRequest) (res *runtime.CheckpointContainerResponse, err error) {
	// The next line is just needed to make the linter happy.
	containerCheckpointTimer.WithValues("no-runtime").UpdateSince(time.Now())
	return nil, status.Errorf(codes.Unimplemented, "method CheckpointContainer not implemented")
}

curl -s https://raw.githubusercontent.com/containerd/containerd/main/internal/cri/server/container_checkpoint.go | wc -l # 34

Finding — this rules out kind, and most clusters

containerd is the default runtime for kind, for most managed Kubernetes offerings, and for most self-managed clusters. On all of them the kubelet checkpoint endpoint exists, passes authorization, and then fails at the runtime with Unimplemented. No configuration changes this.

A secondary source read during this research asserted that container-level checkpoint support “is already available” in containerd. The file above is the whole implementation on main. Check the runtime before planning around this feature.

CRI-O’s implementation, by contrast, is real — and its first act is to check whether support was switched on:

cri-o@main · server/container_checkpoint.gorung 2 · read the source

func (s *Server) CheckpointContainer(ctx context.Context, req *types.CheckpointContainerRequest) (*types.CheckpointContainerResponse, error) {
	if !s.config.CheckpointContainerEnabled() {
		return nil, errors.New("checkpoint/restore support not available")
	}
	...

Section 04

Refusal one: CRI-O ships CRIU support off

Installing CRIU on the node is not enough. CRI-O has its own switch, and it defaults to false.

crio --help on the lab noderung 1 · measured

--enable-criu-support    Enable CRIU integration, requires that the criu binary is
                         available in $PATH. (default: false)  [$CONTAINER_ENABLE_CRIU_SUPPORT]

Switching it on is a drop-in file and a restart. The journal confirms it took:

/etc/crio/crio.conf.d/10-criu.conf, then journalctl -u criorung 1 · measured

[crio.runtime]
enable_criu_support = true
default_runtime = "runc"

crio: "Updating config from drop-in file: /etc/crio/crio.conf.d/10-criu.conf"
crio: "Checkpoint/restore support enabled"

Section 05

Refusal two: the default OCI runtime cannot checkpoint

With the gate on, RBAC granted and CRIU installed, the call still failed — with an error that points at the OCI runtime rather than at CRIU.

checkpoint attempt with CRI-O’s default runtimerung 1 · measured

checkpointing of default/counter/app failed (rpc error: code = Unknown desc =
failed to checkpoint container 2bf13488d06c...: configured runtime does not support checkpoint/restore)

HTTP 500

cri-o@main · internal/oci/runtime_oci.go:1928rung 2 · read the source

func (r *runtimeOCI) checkpointRestoreSupported(runtimePath string) error {
	if err := criu.CheckForCriu(criu.PodCriuVersion); err != nil {
		return fmt.Errorf("check for CRIU %w", err)
	}

	if !crutils.CRRuntimeSupportsCheckpointRestore(runtimePath) {
		return errors.New("configured runtime does not support checkpoint/restore")
	}

	return nil
}

That second check probes the OCI runtime binary. The CRI-O static bundle ships two, and defaults to the one that fails the probe:

asking each bundled runtime directlyrung 1 · measured

$ /usr/libexec/crio/crun checkpoint --help
unknown command checkpoint                        crun version 1.29.1

$ /usr/libexec/crio/runc checkpoint --help
NAME:
   runc checkpoint - checkpoint a running container       runc version 1.5.1

$ sudo crio config | grep default_runtime
default_runtime = "crun"                             <- the shipped default

Finding — the default is the one that cannot

crun can support checkpointing when built against libcriu; the binary in this bundle was not. Since CRI-O defaults to crun, a stock install fails even with everything else correct. Setting default_runtime = "runc" is what made the call succeed.

The error text blames “the configured runtime”, which is accurate but easy to misread as CRI-O itself. It means the OCI runtime — the binary CRI-O shells out to.

Section 06

And CRIU may not be installable at all

Before any of the above, CRIU has to exist on the node. On this platform it could not be installed from the distribution.

Ubuntu 24.04.5 LTS, aarch64, universe enabledrung 1 · measured

$ apt-cache policy criu
criu:
  Installed: (none)
  Candidate: (none)

$ apt-cache search criu
golang-github-checkpoint-restore-go-criu-dev - CRIU bindings for Golang
(no criu binary package)

CRIU 4.2.1 was therefore built from source for this lab. Two dependencies beyond the documented list were required — uuid-dev and libaio-dev — and make install fails at man-page generation without asciidoc, so the binary is installed directly. The set README carries the exact sequence.

the built binary, verifiedrung 1 · measured

$ sudo criu --version
Version: 4.2.1

$ sudo criu check
Looks good.

$ sudo criu check --all
Warn (criu/cr-check.c:824): Dirty tracking is OFF. Memory snapshot will not work.
Warn (criu/cr-check.c:1259): Do not have API to map vDSO - will use mremap() to restore vDSO
Warn (criu/cr-check.c:1179): CRIU built without CONFIG_COMPAT - can't C/R compatible tasks
Looks good but some kernel features are missing

Keep the dirty-tracking warning for chapter 06

criu check passes, and everything in this set works. But dirty page tracking is off on this kernel, which is precisely the mechanism iterative pre-copy migration depends on. Nothing in the upstream Kubernetes path uses it — which is part of why upstream checkpointing is not live migration.

Section 07

Closing note — what varies, and what was not verified

Durable versus run-specific

Durable: containerd returning Unimplemented; the three-field CRI message; CRI-O’s three-field options struct; --enable-criu-support defaulting to false; crun-without-libcriu answering unknown command checkpoint; the exact error strings.
Run-specific: the version numbers, the container ID in the error, and whether your crun build has checkpoint support — that is a build-time choice, not a property of crun.

Not verified

containerd was not run. Its refusal is read from source (rung 2). No containerd cluster was stood up to observe the Unimplemented error in situ.
A crun built with libcriu. Only the bundle’s build was tested. A distribution crun with checkpoint support would likely pass the probe; that was not confirmed.
The pre-1.30 gate-off case. The lab runs 1.36, so the “route not registered” refusal in the figure is inferred from the route-registration source, not observed.
Other CRI implementations. Only containerd and CRI-O were examined.

Section 09

Spoken drills

A team wants to use forensic checkpointing. Their clusters run containerd. What do you tell them?

A strong answer hits

It cannot work — containerd’s CRI implementation returns Unimplemented
The whole file is 34 lines; there is no flag or config that enables it
The kubelet endpoint will still exist and still pass authorization, so the failure appears late
CRI-O is the runtime that implements it today
The honest half: containerd supports checkpointing through its own API; what is missing is the CRI plumbing the kubelet uses. So “containerd can checkpoint” and “Kubernetes can checkpoint on containerd” are different claims

check against §03

You get configured runtime does not support checkpoint/restore. Where do you look?

A strong answer hits

Not CRI-O — the OCI runtime it shells out to
Probe it directly: runc checkpoint --help versus crun checkpoint --help
CRI-O’s bundle defaults to crun, whose shipped build answers unknown command checkpoint
Fix is default_runtime = "runc"
The honest half: the same function also checks for CRIU itself, so this error can equally mean CRIU is missing or too old — read which of the two branches fired

check against §05

Why can you not pass CRIU options — say --tcp-established — through the kubelet API?

A strong answer hits

The CRI request carries only container ID, location and timeout
CRI-O’s options struct carries only Keep, KeepRunning and TargetFile
There is no field anywhere in the chain to hold a CRIU flag
The honest half: this is a deliberate narrowing for the forensic use case, not an oversight — but it means whole classes of workload cannot be checkpointed through Kubernetes even though CRIU itself could handle them

check against §02

You install CRIU on every node and nothing changes. What did you miss?

A strong answer hits

CRI-O gates CRIU behind --enable-criu-support, default false
Even then the default OCI runtime may not support it
Confirm with the journal line “Checkpoint/restore support enabled”
The honest half: on some platforms CRIU is not packaged at all — Ubuntu 24.04 arm64 has no criu candidate, so “install CRIU” may itself mean building from source

check against §04 and §06

Four components, four ways to be told no

What actually crosses the CRI

containerd answers the call and declines it

Refusal one: CRI-O ships CRIU support off

Refusal two: the default OCI runtime cannot checkpoint

And CRIU may not be installable at all

Further reading

Closing note — what varies, and what was not verified

Durable versus run-specific

Not verified

Spoken drills