CLONE Dump Design Document

Covers the experimental CLONE dump path in this CRIU fork (--clone-dump).

Overview

CLONE dump is an experimental CRIU mode designed to minimize source-process downtime during live migration. It uses Linux's userfaultfd write-protect (UFFD_FEATURE_WP_ASYNC) and PAGEMAP_SCAN to track memory writes while the process keeps running, reducing the freeze time so pages can stream to the target while the source continues making progress.

Goals

Reduce downtime at the source. The source process must be responsive almost the entire migration; only two short freezes are acceptable.
Minimize total migration time. The wall-clock from "start migrating" to "process running on target" should be dominated by physical resources — network bandwidth or CPU — not by design limitations.
No disk needed. Pages stream directly from source memory to target memory; no intermediate image file is written for pages.

Top-Level Architecture

CLONE dump moves a running process from one host (the source) to another (the target) by trading the long single-shot freeze of a classic criu dump for two short freezes that bracket a long, online page-streaming phase. The sketch below walks through the flow phase by phase, with the source on the left and the target on the right; the two sides are connected by a fan of TCP sockets that carry compressed page batches.

Source Side (dump host)

Target Side (restore host)

⇄ Per-thread TCP sockets (N connections) ⇄

The source briefly freezes the process, registers all of its writable VMAs with userfaultfd in async write-protect mode, and applies the write-protect in parallel across the address space. As soon as that's done the process is unfrozen and resumes execution; from this point on every page it writes leaves a soft-dirty trail that the kernel exposes through PAGEMAP_SCAN. The target has nothing to do yet — it just waits for the source to open the per-thread page-transfer connections.

Seize + dirty-tracking init

Seize the task tree, open a write-protect userfaultfd inside the target process via the parasite, apply WP across the address space in parallel, then unfreeze.

Waiting for the per-thread page-transfer connections.

↓

With the process running, the source ships every tracked page once. A pool of bulk-sender threads pulls work items off a shared queue, reads each chunk out of the live process with process_vm_readv, compresses it, and writes it onto its own TCP socket. On the target a matching pool of receiver threads decompresses each batch and parks it in an in-memory buffer keyed by virtual address. Nothing is applied to the target process yet — pages only accumulate. Only a subset of the configured sender threads run during the bulk pass (controlled by --clone-p3-threads-bulk); the rest stay parked. The receiver is typically the bottleneck during bulk, so adding more senders just contends for CPU. Scanners exist on the source but are also blocked on a barrier; they cannot start their iterative work until the bulk pass has covered the whole address space.

Bulk transfer

Bulk-sender threads work-steal chunks of address space from a shared queue, read them with process_vm_readv, compress with LZ4, and send on a per-thread socket.

Scanner threads exist but are parked on a barrier until the bulk pass has covered every tracked VMA.

Receivers

One receiver thread per source connection. Each one decompresses incoming batches and inserts them as fresh entries in the page buffer.

↓

Page buffer

A hash table keyed by virtual address; pages accumulate here for the rest of Phase 2 and Phase 3 without being applied to the target process.

No drain, no restore yet.

↓

Phase 2b is optional and off by default — it's enabled with --clone-pre-scan. When disabled, the source skips straight to the Phase 3 freeze and does a single final dirty scan there. When enabled, Phase 2b runs an extra convergence loop while the process is still running: the bulk pass shipped one snapshot in time, so by the time it finishes the running process has already dirtied a fresh set of pages. Scanner threads ask the kernel for the set of write-protected-but-touched pages via PAGEMAP_SCAN, publish those dirty regions to a shared queue, and the same sender threads that did the bulk pass — now all of them — drain that queue and ship the new dirty pages. On the target the receivers detect that a re-sent region already lives in the page buffer and decompress straight on top of the older copy in place — no allocation, no memcpy. Scanners loop until the dirty-page count has dropped below a convergence threshold (or a max-iteration cap is hit).

Iterative dirty scan

Scanners ask the kernel for write-protected-but-dirty pages and publish the dirty regions to a lock-free MPMC queue.

↓

Pack → read → compress → send

The full sender pool drains the queue, packs regions into batches, and pushes them out the same way as in the bulk pass.

In-place overwrite

Receivers detect that the address range is already in the buffer and decompress on top of the older copy in place — no allocation, no memcpy.

Still no drain, still no restore.

↓

This is the second and final freeze on the source — and the only meaningful downtime the user sees. The source re-seizes the task tree, runs one last dirty-page scan to catch anything written between the last Phase 2b iteration and the freeze, sends those pages, detects any VMAs that appeared after Phase 1 and ships those too, and then writes a "skeleton" CRIU image that contains the whole process state except page contents (which are already on the target). The target is still just buffering — drain is deliberately deferred until after this freeze ends.

Freeze + final flush + skeleton dump

Re-freeze, run a final dirty scan, send any pages from VMAs that appeared after Phase 1, then write the skeleton image (everything but page contents).

Receivers keep buffering — drain has not started yet.

↓

While still frozen the source sends every skeleton image file, then signals "all pages sent". As soon as that signal is on the wire the source unfreezes its task tree (or kills it if migration is one-way), and only after unfreezing does it wait for the target's ACK — that wait is off the critical path. On the target, criu restore takes over: it builds the task tree from the skeleton image, freezes the new tasks, and only then starts the drain threads that apply the buffered pages via UFFDIO_COPY. Drain and restore are deliberately serial — restore waits for the buffer to empty before it lets the new tasks run, so from the restored process's point of view its memory is fully populated when it resumes.

Signal, unfreeze, clean up

Send the "all pages sent" marker, unfreeze the tasks, then wait for the target's ACK off the critical path and tear down the userfaultfd and TCP connections.

Restore takes over

The target sends an immediate ACK, then runs criu restore against the skeleton image. New tasks are built and frozen.

↓

Drain

Drain threads walk the page buffer and apply each batch via UFFDIO_COPY, returning the underlying memory to the page pool as buffer chunks empty.

↓

Unfreeze

Once the buffer is empty, restore unfreezes the new tasks. From the restored process's point of view, the migration is indistinguishable from a normal CRIU restore.

TLS Support

CLONE dump can be deployed across an untrusted network — every page of the migrated process flows over TCP — so each connection in the per-thread fan can be wrapped in TLS. The implementation lives in tls-conn.c and uses GnuTLS. A single SSL_CTX-style credentials object is built once at start-up from the same certificate / key / CA paths that criu already uses for its existing --tls flag, and each individual P3 socket gets its own session that reuses the shared credentials.

TLS is optional and orthogonal to everything else in the design. When the user runs criu with --tls, both the source and target wrap their P3 sockets transparently; when it's off, the same code paths fall back to raw TCP. The compression and packing logic above the socket is identical either way — TLS sees a stream of compressed page batches and treats them as opaque bytes.

Compression Support

Every page batch on the wire is LZ4-compressed. Compression happens in the sender thread immediately before send(), decompression happens in the matching receiver thread immediately after recv(). The wire format is the standard page_server_iov header followed by a four-byte compressed_size and that many bytes of LZ4 payload — the receiver knows the uncompressed size from nr_pages × PAGE_SIZE.

Compression is on a per-batch basis (one LZ4 frame per batch), not a streaming compressor across the whole connection. That keeps each receiver decompression independent — there is no shared state to coordinate between threads — and lets the receiver decompress straight into its destination buffer (the page-pool slot in Phase 2a, or directly on top of an existing entry in Phase 2b's overwrite path).

The LZ4 acceleration parameter is set to 1 on every code path. Higher acceleration values trade ratio for CPU and were measured to regress total wall-clock: a larger compressed payload pushed the bottleneck from the sender's CPU into tcp_sendmsg and the kernel's skb_page_frag_refill path. Aggregate compression ratio is exposed as a runtime statistic so it can be sanity-checked per workload.

Handling New VMAs

The set of VMAs registered with userfaultfd at the end of Phase 1 is a snapshot. The process is then running for the entirety of Phase 2 — possibly minutes — and is free to call mmap(), mprotect(), brk(), or otherwise extend its address space during that window. Any region created after Phase 1 is not write-protected, so its writes leave no soft-dirty trail and the scanners would simply miss it.

To catch this, Phase 3 (with the process re-frozen) re-collects the live VMA list and compares it to the set tracked at Phase 1. Anything new — a freshly-mapped region, an extension of an existing VMA, or a region that was unmapped and then re-mapped at the same address — is treated as fully dirty: every page in it is queued for transfer regardless of pagemap state. The new ranges are also appended to the lazy-VMA list so the senders pick them up alongside the final dirty-page batch from the convergence scan, and so the target's page buffer keys match.

The combined effect is that the post-restore process sees the address space the source had at the moment of the final freeze rather than the one it had at Phase 1. The cost is a single VMA-list diff and one extra batch of full-region transfers during the Phase 3 freeze, proportional to how much new mapping the process did during Phase 2 — which for most workloads is zero.

Unmap Support

The mirror problem to new VMAs is regions that disappear. While the process is running it can munmap(), mremap(), or hit any other path that drops a mapping. If CLONE dump kept ploughing through with a stale VMA list, two things would go wrong: (1) the source-side scanner would issue PAGEMAP_SCAN against an address range that no longer exists and waste work or hit errors; (2) the target's page buffer would still hold pages for that region and try to apply them after restore, which on the new task tree would either install garbage or hit EEXIST / ENOENT.

Two complementary mechanisms catch unmaps. The first is a kernel-level signal: userfaultfd delivers a UFFD_EVENT_UNMAP / UFFD_EVENT_REMOVE for any range that disappears under the registered uffd. A dedicated reader thread on the source consumes those events and records the unmapped ranges. The second is a fallback: if the scanner-then-sender path tries to process_vm_readv a region that has just been unmapped, the syscall returns EFAULT, and the sender records the range exactly as if a UFFD event had fired. The two paths converge on a single unmapped-range set held by the dump driver.

At Phase 3, after the process is re-frozen and the new-VMA pass has run, that unmapped-range set is folded into the decision logic. Any region marked as unmapped that no longer has a live VMA is dropped from the lazy-VMA list and from any pending sender work. Regions that were unmapped and then re-mapped at the same address are treated by the new-VMA pass as fresh, so they get a full transfer. On the target, the page buffer entries for genuinely unmapped ranges are dropped before drain so UFFDIO_COPY never tries to apply them.

Usage and Configuration

CLONE dump is selected with the --clone-dump flag on the source. The target uses a dedicated criu clone-receive mode that combines the page-server, lazy-pages daemon, and restore into one process — once it has all pages it triggers the restore itself, no separate criu restore step is required.

Minimal example

# target host
sudo criu clone-receive \
    --images-dir /tmp/img \
    --address <source-ip> --port 27 -v4

# source host
sudo criu dump -t <pid> \
    --images-dir /tmp/img \
    --clone-dump \
    --page-server --address <target-ip> --port 27 -v4

Runtime tunables

Defaults are sized for live-migrating multi-gigabyte processes; smaller workloads can usually leave them alone.

Option	Default	Effect
`--clone-dump`	off	Enables the phased dump path on the source. Required for everything below.
`--clone-p3-threads N`	15	Total page-sender threads on the source (and matching receiver threads on the target).
`--clone-p3-threads-bulk N`	15	How many of those senders run during the bulk pass. Lower than `--clone-p3-threads` when the receiver is the bottleneck.
`--clone-scanners N`	20	Source-side scanner threads issuing `PAGEMAP_SCAN`.
`--clone-pre-scanners N`	1	Subset of scanners active during the optional Phase 2b pre-scan loop.
`--clone-drain-threads N`	20	Target-side threads applying buffered pages via `UFFDIO_COPY`.
`--clone-pre-scan`	off	Enables the Phase 2b iterative pre-freeze convergence loop. When off, the source skips straight to the Phase 3 freeze.
`--tls`	off	Wraps every per-thread P3 socket in a TLS session (uses CRIU's existing TLS configuration).

Kernel and system requirements

Linux 6.7+ on both source and target — needs both UFFD_FEATURE_WP_ASYNC (kernel 6.1+) and PAGEMAP_SCAN (kernel 6.7+).
vm.unprivileged_userfaultfd=1 on both hosts.
TCP reachability from source to target.

Known Limitations

Single process tree: tracking metadata is shared-global, so only one CLONE dump can run at a time.
Kernel requirements: Linux 6.7+ — requires both UFFD_FEATURE_WP_ASYNC (kernel 6.1+) and PAGEMAP_SCAN (kernel 6.7+).
UFFD cleanup: page-table walks during UFFDIO_UNREGISTER can take seconds on large memory; cleanup is chunked and relies on close-on-exit for the rest.

Future Improvements

Multi-process tree support (drop global singletons).
Adaptive convergence threshold driven by measured write rate.
WP_SYNC support — switch the dirty-tracking path from UFFD_FEATURE_WP_ASYNC to synchronous write-protect during the convergence phase, so writes to already-shipped pages block until CRIU has re-fetched them. This bounds the worst-case dirty set during Phase 2b and shortens convergence on write-heavy workloads at the cost of brief in-process stalls.