The Kubelet Checkpoint API  ·  Chapter 05  ·  measured, not recalled

The Archive Is Memory

A value delivered to a pod through a Kubernetes Secret appeared zero times in the container’s logs and zero times in its pod spec. It appeared twice inside the checkpoint archive — once in the dumped OCI spec and once in the memory pages — and came back out with a single strings | grep. A checkpoint file is a memory dump with a tar extension, and it should be handled like one.

measured on Kubernetes v1.36.4 · CRI-O 1.36.5 · CRIU 4.2.1 · kernel 6.8.0-139-generic
secret supplied via secretKeyRef, never written to a file by the workload
mode A — the canary is synthetic, generated from /dev/urandom on each run; the block below is one run, 2026-09-15, so your value and archive size will differ

Section 01

The experiment

A pod receives a secret the way most applications do: a Kubernetes Secret, projected into an environment variable via secretKeyRef. The workload reads it into a variable at startup and then does nothing with it — never logs it, never writes it to disk.

Before checkpointing, the value is confirmed absent from the two places an operator would look.

before the checkpointrung 1 · measured
occurrences in pod logs   : 0
occurrences in pod spec   : 0     (secretKeyRef, so the value is not in the manifest)

Then one call to the kubelet checkpoint endpoint.

Section 02

Recovering it

the archive, and one greprung 1 · measured
archive mode=600 owner=root:root bytes=969216

$ sudo tar xf checkpoint-secretpod_default-app-*.tar -C /tmp/xt
$ sudo grep -rl "$SECRET" /tmp/xt
spec.dump
checkpoint/pages-1.img

$ sudo strings /tmp/xt/checkpoint/pages-1.img | grep -o "CANARY-[a-z0-9-]*"
CANARY-24ba3812cfcf82b8-synthetic
bash demos/scripts/secret-in-archive.sh
Finding

No exotic tooling. tar, strings, grep. The value was never on disk inside the container and never in a log, and it is still trivially recoverable from the archive, because the archive contains the process’s anonymous memory verbatim.

Everything the process had in memory is in there: decrypted payloads, session tokens, private keys it loaded, the contents of every Secret it read, and anything a user typed into it.

Secret object RBAC-protected env var secretKeyRef process memory held in a variable where an operator looks pod logs: 0 pod spec: 0 one POST the checkpoint archive — mode 0600, root:root checkpoint/pages-1.img ← the memory itself spec.dump ← the container’s env recoverable with: strings | grep the Secret’s RBAC protects the object. Nothing carries that protection into the archive.
The access control does not travel with the data. Reading the Secret object requires RBAC on Secrets. Reading the same bytes out of a checkpoint requires only the ability to read a file on a node — a completely different, and usually much larger, set of people.

Section 03

Two places, for two different reasons

  • checkpoint/pages-1.img — the process’s anonymous memory, dumped verbatim. This is unavoidable: capturing memory is the entire point of a checkpoint. Any secret a process has ever loaded and not overwritten is a candidate.
  • spec.dump — the container’s dumped OCI specification, which includes its environment variables. This one catches people out, because it means a secret is in the archive even if the process never read the variable.
A consequence for the restore path

Because spec.dump carries the environment, the OCI image you build in chapter 03 to restore a container also carries it. If that image is pushed to a registry — which is the natural way to move a checkpoint between nodes — the secret goes with it, into a system that usually has broader read access than either the node or the Secret.

Section 04

Who can create one, and who can read one

Two separate questions, with two separate answers, and the gap between them is the interesting part.

ActionWhat it requiresTypically held by
Create a checkpointcreate on the nodes/checkpoint subresource, via the kubelet’s authorization webhookNobody by default — must be granted explicitly (chapter 01)
Read the archiveRead access to a 0600 root:root file under /var/lib/kubelet/checkpoints/Anyone with root on the node: node admins, privileged DaemonSets, anything that can mount the host filesystem
Read the Secret it containsNothing further — it is plain bytes in the fileAs above
Finding — the permission that matters is not the API one

The RBAC on nodes/checkpoint is a real control and it is off by default, which is good. But it governs creation. Once an archive exists it is an ordinary file, and its protection is filesystem permissions on the node — which a privileged pod with a hostPath mount already has.

So the exposure is: a workload that could not read a Secret through the API can read it out of a checkpoint of a pod that could. Checkpoints should be treated as an escalation path, and cleaned up rather than left lying in /var/lib/kubelet/checkpoints/.

Section 05

This is what “forensic” was telling you

The feature’s name is not decoration. KEP-2008 is Forensic Container Checkpointing, and forensics is the discipline of extracting evidence from a system without the system’s cooperation. A tool built for that purpose is, by construction, a tool that extracts everything.

Chapter 01 showed the checkpoint is non-destructive — the container keeps running and nothing in it is notified. That is also a forensic property: the point is to take a copy without tipping off whatever is inside. It is worth stating plainly that the same property makes the operation invisible to the workload’s own audit trail.

Where it does leave a trace

The request is authorized through the API server, so a create on nodes/checkpoint is an auditable event in the API server’s audit log, and CRI-O logs Checkpointing container: <id> to the node journal. Both were observed on this lab. The workload sees nothing; the cluster does — provided audit logging is on and the node journal is shipped.

Section 06

What to actually do about it

  • Treat the grant as sensitive. create on nodes/checkpoint is effectively “read the memory of any pod on this node”. Scope it, and prefer granting it temporarily for an investigation.
  • Treat the archive as a memory dump. Same handling as a core dump: restricted storage, short retention, deliberate deletion. It is not a backup artifact.
  • Do not push checkpoint images to shared registries without accepting that the environment and memory travel with them.
  • Audit the creation, not the file. The API-server audit event is the reliable signal; the file appearing on a node is not something the cluster will tell you about.
  • Remember it works on any pod on the node, not only ones you own. Namespace boundaries do not constrain it — the subresource is on the node.

Section 07

Further reading

articleForensic container analysiskubernetes.io · 2023-03-10 · analysing an archive — the intended use of what §02 does toolcheckpointctlinspect archives properly instead of by grep primaryCRIU image formatwhat pages-*.img actually is docsKubernetes Secrets — security propertieswhat the Secret guarantees, and where those guarantees end docsAuditingwhere the checkpoint request shows up KEPKEP-2008the security considerations section is worth reading alongside this chapter

Section 08

Closing note — what varies, and what was not verified

Durable versus run-specific

  • Durable: that the value is absent from logs and spec but present in pages-1.img and spec.dump; that mode is 0600 root:root; that plain strings recovers it.
  • Run-specific: the byte count (870,400), the canary string itself, and which numbered pages image holds it — that depends on the process’s memory layout.

Scope and honesty about the test

  • The canary is synthetic, generated for this experiment. No real credential was used anywhere in this set.
  • The workload holds the secret in a live variable and never zeroes it. A program that overwrites a secret after use would reduce, though not reliably eliminate, its presence in the dump — memory may still hold copies. That was not tested.
  • Only an environment-variable secret was tested. A secret mounted as a file would additionally appear in rootfs-diff.tar or via the mount; not measured.

Not verified

  • Encryption at rest. Nothing here encrypts the archive, and no option to do so was found in the chain examined in chapter 02. Absence of evidence — it was not exhaustively searched.
  • The audit event itself. API-server audit logging was not enabled on this lab; the claim that the request is auditable follows from it being an authorization decision at the API server, and from the observed CRI-O journal line. The audit record was not inspected.
  • Whether any policy engine blocks it. Admission controllers do not see this — it is a kubelet endpoint, not an API-server write. Not tested against a policy engine.

Section 09

Spoken drills

A platform engineer asks for checkpoint permission so they can debug a crashing pod. What do you need to tell them before granting it?

A strong answer hits

  • The grant is on nodes/checkpoint — it covers every pod on the node, not just theirs
  • The resulting archive contains process memory, so any Secret those pods loaded is in it
  • The environment is also captured in spec.dump, even if unread by the process
  • Scope and time-bound the grant; treat archives like core dumps
  • The honest half: it is a genuinely good debugging tool and refusing outright is not the answer — the ask is legitimate, the blast radius just is not what the requester thinks

check against §04

Your secrets are in a Kubernetes Secret with tight RBAC. Does checkpointing change your exposure?

A strong answer hits

  • Yes — the RBAC protects the object, not the bytes once they reach a process
  • Measured: 0 occurrences in logs and spec, 2 in the archive, recovered with strings
  • Reading the archive needs node root, not Secret RBAC — a different and usually wider group
  • The honest half: this is not unique to checkpointing. A core dump or a node-level memory read has the same property. What is new is that Kubernetes now has an API that produces one on demand

check against §02 and §04

Why does the environment end up in the archive even for a process that never reads it?

A strong answer hits

  • spec.dump is the container’s OCI spec, and env vars are part of it
  • So it is captured as configuration, independently of memory
  • That means it also rides along in any restore image built from the archive
  • The honest half: this is necessary — a restore has to recreate the container’s configuration, and the environment is part of that. It is a real consequence, not a bug to be fixed

check against §03

Can a workload tell that it has been checkpointed?

A strong answer hits

  • No — it is frozen and thawed; no restart, no signal, no event delivered to it
  • That invisibility is deliberate; the use case is forensic
  • The cluster can see it: an authorization decision at the API server, and a CRI-O journal line
  • The honest half: the process does experience a pause, so a workload watching its own clock could infer something happened — but nothing tells it what

check against §05