COW Dump Design Document

Generated from COW_DUMP_DESIGN.md. Covers the experimental COW dump path in this CRIU fork (--cow-dump).

Overview

The COW (Copy-on-Write) dump implementation is an experimental feature in this CRIU fork designed to minimize source process downtime during live migration. Instead of freezing the process for the entire dump duration, COW dump uses Linux's userfaultfd write-protect (UFFD_FEATURE_WP_ASYNC) to track memory writes while the process continues running, enabling incremental page transfer.

Goals

1. Minimize downtime: Keep the source process running during the bulk of the memory transfer; only freeze for a short skeleton dump + final dirty flush.
2. Parallel, compressed transfer: Multiple sender/receiver sockets, LZ4 compression, batched I/O.
3. Convergence: Iteratively re-send dirty pages until a threshold is reached (overwriting older copies in the replica batch buffer), then do a single final frozen scan.
4. Completeness: Capture dynamically created VMAs as well as the originally-tracked set.

Top-Level Architecture

Primary and replica are shown side-by-side so you can read horizontally to see what each side is doing in each phase.

Phases

Key Data Structures

Dump-side state — struct cow_dump_info (cow-dump.c)

Tracks the active dump: source pid, destination id (valid only after Phase 3 pstree collection), the WP_ASYNC uffd fd returned by the parasite, and the list of tracked VMAs (which grows when new VMAs are detected in Phase 3). A phase enum cycles IDLE → ASYNC_BULK → SCAN → DONE.

Lazy VMA — struct lazy_vma_entry (cow-mem.c)

One per tracked VMA on the dump side: [start, end) range, back-pointer to the vma_area (NULL for regions discovered in Phase 3), total page count, and the dst_id / source_pid that identify which task the VMA belongs to. Entries form a global list used by scanners and bulk senders.

Dirty region — struct dirty_region_entry (cow-bulk-send.h)

Minimal record pushed from scanners into the convergence queue: the [start, end) dirty range plus dst_id / source_pid. Senders read these and issue process_vm_readv to grab the pages.

MPMC convergence queue (cow-bulk-send.c)

A flat 8M-slot array (CONV_QUEUE_CAP) of region pointers with two atomic cursors. Scanners produce by fetch-add on the head and store their region pointer in the claimed slot. P3 senders consume by CAS on the tail — one region per successful CAS, then a short busy-read until the slot's pointer becomes visible. No empty-queue polling waste and no per-slot lock.

Replica batch buffer — struct batch_buffer_entry (cow-uffd.c)

One entry per 1MB-aligned region of the target process: a data pointer into the page pool (256 contiguous 4KB pages), a page_bitmap tracking which of the 256 pages have arrived, plus a secondary bitmap for drain accounting.

• Primary index: hash table, COW_BATCH_BUFFER_HASH_SIZE = 256K buckets, one spinlock per bucket.
• Secondary index: chunk_index[COW_MAX_POOL_CHUNKS] maps each page-pool chunk to the list of batches that came out of it — drain can free chunks progressively rather than waiting for the whole buffer to empty.

Page pool (page-pool.c)

• Chunk = COW_CHUNK_SIZE (64MB), chunk-aligned, max COW_MAX_POOL_CHUNKS (8192) ⇒ 512GB ceiling.
• Per-thread bump allocator (COW_MAX_THREADS slots); ownership transfer via page_pool_get_pages() / page_pool_put().
• Atomic refcount per chunk header (page 0 of each chunk). munmap() once refcount drops to 0.

Configuration

All tunables live in criu/include/cow/cow-conf.h. Key constants:

Thread counts are profile-gated:

Notable compile-time feature flags (also in cow-conf.h):

• COW_PRE_SCAN — iterative pre-freeze scan (on by default).
• CONFIG_PAGE_STATE_TRACKER, CONFIG_HUNG_PAGE_TRACKER, CONFIG_COW_COMPARE, CONFIG_COW_COMPARE_PAGES — diagnostics, off by default.

Protocol

Three distinct message channels are in use:

1. TCP Page-server TCP — PRIMARY (dumper) ↔ REPLICA (lazy-pages daemon), over the network. One main control socket plus COW_NUM_P3_THREADS parallel P3 data sockets. Frames are struct page_server_iov headers with optional trailing payload.
2. UNIX Lazy-pages UNIX socket — REPLICA lazy-pages daemon ↔ REPLICA criu restore, same host. Carries fixed-size uint32_t signal values.
3. RPC Parasite compel RPC — criu dump ↔ parasite thread injected into the target process (same host). Used during Phase 1 only.

TCP Page-server messages (cow-page-xfer.h)

Four message types are actively used on the page-server TCP channel. All share the same struct page_server_iov header; only PS_IOV_ADD_F_COMPRESS carries a trailing payload.

Reserved/not used in the COW path:

• PS_IOV_ADD_F_PF (9) — placeholder for per-fault push; current COW flow uses PS_IOV_ADD_F_COMPRESS batches exclusively.
• PS_IOV_START_RESTORE (12) — defined in the header, never sent on the wire in the COW path.

PS_IOV_ADD_F_COMPRESS wire format (cow-bulk-send.c:send_pages_batch_compressed): the 24-byte page_server_iov header carries the encoded cmd, nr_pages (≤ COW_BATCH_PAGES = 256), base vaddr (may not be batch-aligned), and dst_id. Immediately after it comes a 4-byte compressed_size followed by that many bytes of LZ4-compressed page payload.

LZ4 acceleration: 1 on both the pre-freeze bulk path and the frozen convergence/new-VMA path. An experiment with acceleration=99 during convergence regressed total P3 wall-clock from 2.3s → 4.8s (larger wire size shifted the bottleneck to tcp_sendmsg/skb_page_frag_refill).

UNIX Lazy-pages UNIX socket signals (criu/uffd.c)

Single-direction uint32_t cookie values over the in-host UNIX socket between the lazy-pages daemon and criu restore. Two cookies (TASKS_FROZEN, DRAIN_COMPLETE) are COW-only; RESTORE_FINISHED exists in the non-COW path as well.

RPC Parasite compel RPC (criu/pie/parasite.c)

Per-phase sequence

Three swimlanes show who sends what in each phase. Channel tags: TCP page-server, UNIX lazy-pages socket, RPC parasite compel.

Soft Error Handling (UFFDIO_COPY on replica)

Wrapper: cow_uffd_copy_pages() / cow_uffd_copy() in cow-uffd.c.

Flags: COW_TRACK_STRICT (BUG on EEXIST/ERROR, used during drain), COW_TRACK_RETRY (retry mode, returns -EAGAIN instead of queueing).

File Organization

criu/cow/ — C implementations

criu/include/cow/ — headers

cow-conf.h (central tunables), cow-dump.h, cow-mem.h, cow-bulk-send.h, cow-bulk-recv.h, cow-page-xfer.h, cow-unified-thread.h, cow-uffd.h, cow-lazy-pages.h, cow-compare.h, cow-batch-bitmap.h, cow-bitmap.h, page-pool.h, page-state-tracker.h, hung-page-tracker.h, unmapped-tracker.h, pf-tracker.h, spsc-queue.h, spmc-queue.h, mpsc-queue.h.

mpsc-queue.h and spsc-queue.h are kept for the page-request queue (cow-unified-thread.c) and legacy interfaces. The scanner→sender hot path now uses the flat-array MPMC convergence queue defined directly in cow-bulk-send.c.

Integration Points

criu/cr-dump.c

• cr_dump_tasks() redirects to cr_dump_tasks_cow_phased() when opts.cow_dump && opts.lazy_pages.
• Inside cr_dump_tasks_cow_phased():
  • Phase 1: pre_dump_one_task loop, pstree_switch_state(TASK_ALIVE).
  • Phase 2a + 2b: cr_page_server(false, true, -1) (which internally invokes cow_dump_init_async + parasite RPC earlier, then starts P3 threads via the unified thread). Bulk senders exhaust g_work_queue first (Phase 2a); scanners then wake and iterate PAGEMAP_SCAN/convergence-queue sends (Phase 2b). The loop while (!cow_all_threads_below_threshold()) usleep(10ms) holds until both complete.
  • Phase 3: reseize_pstree, collect_pstree_ids, cow_set_dst_id, collect_mappings + cow_detect_new_vmas + cow_set_new_vma_ranges, cow_signal_last_scan, skeleton dump_one_task loop, cow_wait_p3_threads, cow_free_new_vma_ranges, cow_set_phase(COW_PHASE_DONE), write_img_inventory.
• cr_dump_finish(): sends PS_IOV_ALL_PAGES_SENT, calls cow_cleanup_async_uffd(), cow_dump_fini().

criu/config.c

Option --cow-dump → opts.cow_dump (case 1105).

criu/mem.c

When opts.cow_dump, generate_iovs() calls cow_mem_add_lazy_vma() to append lazy-capable VMAs to global_lazy_vmas. In Phase 3 skeleton dump mode, it short-circuits page iteration (mdc.cow_skeleton_non_lazy = true).

criu/pie/parasite.c

PARASITE_CMD_COW_DUMP_INIT handler (parasite_cow_dump_init) creates the uffd with the requested features (UFFD_FEATURE_WP_ASYNC) and returns the fd over the compel socket.

Debugging

• TIMING: prefix on wall-clock logs at every phase boundary and major step. TIMING @X.XXXXXX: tags deltas relative to freeze start.
• VMA_TRACE: prefix on VMA-lifecycle logs (registration, scanning, new detection).
• DEBUG_PERF: per-scanner/sender iteration timing.
• check_and_print_uffd_stats() — UFFD fault histogram by batch size (cow_get_histogram_bucket).
• g_compress_uncompressed_bytes / g_compress_compressed_bytes — aggregate compression ratio. Thread-local counters are flushed at thread exit (one atomic per counter, not per-send).
• page-state-tracker.c (off by default) keeps a 16-entry history per page for diagnosing migration divergence.

Usage

# PRIMARY (source)
sudo criu dump -t <pid> -D /images --cow-dump --lazy-pages \
    --page-server --address <replica-ip> --port 27

# REPLICA (destination)
sudo criu page-server --images-dir /images --port 27 --lazy-pages
sudo criu restore   --images-dir /images --lazy-pages

Known Limitations

1. Single process tree: tracking metadata is shared-global (g_cow_info, global_lazy_vmas).
2. Kernel requirements: Linux 5.7+ for UFFD_FEATURE_WP_ASYNC. PAGEMAP_SCAN (kernel 6.6+) or fallback path needed.
3. Memory overhead: replica uses up to 512GB worth of 64MB chunks (COW_MAX_POOL_CHUNKS), refcount-freed as drain progresses.
4. UFFD cleanup: page-table walks during UFFDIO_UNREGISTER can take seconds on large memory; cow_cleanup_async_uffd() chunks the work and relies on close-on-exit for the rest. A hardcoded 15s sleep currently precedes cleanup (investigation outstanding).
5. Network lock skipped: COW path does not call network_lock().
6. New-VMA metadata gap: VMAs created between Phase 1 and Phase 3 have their pages transferred but their VMA records are not re-dumped — they won't exist on the replica. Logged as pr_err at Phase 3.

Future Improvements

1. Multi-process tree support (drop global singletons).
2. Adaptive convergence threshold driven by measured write rate.
3. Better handling of rapidly-dirtying workloads (e.g. pin-to-CPU heuristics during bulk already in COW_P3_SENDER_CPU).
4. Integration with container runtimes (containerd, CRI-O).