Persist snapshot to disk

[ludfjig]

Please review

persist sandbox snapshots to disk

Adds Snapshot::to_file / Snapshot::from_file and MultiUseSandbox::from_snapshot, so a sandbox can be reconstructed from a saved snapshot without going through UninitializedSandbox::evolve(). Works in-process and across processes.

public API

let snapshot = sandbox.snapshot()?;
snapshot.to_file("guest.hls")?;

let snapshot = Arc::new(Snapshot::from_file("guest.hls")?);
let cfg: Option<SandboxConfiguration> = None;
let mut sandbox = MultiUseSandbox::from_snapshot(snapshot, HostFunctions::default(), cfg)?;

HostFunctions must be a superset (by name and signature) of those registered when the snapshot was taken. The optional SandboxConfiguration overrides runtime fields (interrupt knobs, guest_core_dump, guest_debug_info). Layout fields (input_data_size, output_data_size, heap_size, scratch_size) are always taken from the snapshot.

from_file_unchecked is provided for trusted environments. It still verifies the header hash but skips the memory blob hash, making large snapshots load in constant time.

file format

+-----------------+ 0
| RawPreamble     |  magic "HLS\0" + format_version
+-----------------+
| RawHeaderV1     |  arch, abi_version, layout, entrypoint, sregs flag,
|                 |  hypervisor, memory_size, memory_offset, host_funcs_size
+-----------------+
| RawSregs        |  8 segment regs, 2 table regs, 7 control regs,
|                 |  4-element interrupt bitmap. Always written.
+-----------------+
| RawHashes       |  header_hash (always verified) + blob_hash
+-----------------+
| host functions  |  flatbuffer of names + signatures (variable, optional)
+-----------------+ <- memory_offset (page-aligned)
| memory blob     |  memory_size bytes, mmap target
+-----------------+
| trailing guard  |  PAGE_SIZE of zeros (Windows guard backing)
+-----------------+

All header structs are #[repr(C)] POD types deriving bytemuck::Pod + Zeroable, so the byte layout is whatever those structs declare and there are no separate offsets to keep in sync.

The trailing PAGE_SIZE padding exists because Windows read-only file mappings cannot extend beyond the file's actual size, so the file must contain bytes for the trailing guard. Linux ignores the padding (its guard pages come from an anonymous mmap reservation).

versioning and portability tags

• RawPreamble.format_version. Wire format of the file. Bumped when header byte layout or section ordering changes. May be convertible across versions.
• RawHeaderV1.abi_version. ABI of the memory blob contents. Bumped independently when in-guest data layouts change. A mismatch means the snapshot must be regenerated from the guest binary.
• arch tag distinguishes guest arch (x86_64, aarch64, i686), so an i686-guest snapshot cannot be loaded into an amd64-guest build.
• hypervisor tag distinguishes KVM, MSHV, and WHP. Segment register hidden-cache fields (unusable, type_, granularity, db) differ between hypervisors for the same architectural state, so cross-hypervisor loads are rejected by default.

what is and is not persisted

Persisted: the snapshot region (guest code, PEB, heap, init data, page tables), all sregs, and the names + signatures of host functions registered at snapshot time.

Not persisted:

• sandbox_id. Process-local counter, fresh ID assigned on load (see "sandbox identity and restore" below).
• LoadInfo. Debug-only, reconstructible from ELF if needed (see "known limitations").
• regions. Always empty after snapshot construction (mapped-region contents are absorbed into the memory blob, see "known limitations").

The scratch region is recreated fresh on load and re-initialised by copying page tables from snapshot to scratch and writing I/O buffer metadata.

The vCPU special registers are persisted because the guest init code installs a GDT, IDT, TSS, and segment descriptors that differ from the standard 64-bit defaults.

A header_hash over preamble || header || sregs || host_funcs is always verified, even by from_file_unchecked. A separate blob_hash over the memory blob is verified by from_file and skipped by from_file_unchecked. Because blob_hash is itself one of the bytes covered by header_hash, swapping a memory blob without rewriting the header invalidates the always-checked header hash.

gdb / crashdump

• guest_debug_info and guest_core_dump fields of SandboxConfiguration are honored by from_snapshot, so gdb and core dumps work after loading from disk.
• For gdb, HyperlightVm::new installs a one-shot entry breakpoint for both Initialise and Call snapshots so the gdb stub event loop enters on the first vCPU run regardless of how the sandbox was constructed. The breakpoint is removed on first hit by the run loop.

sandbox identity and restore

MultiUseSandbox::restore requires the supplied snapshot to share the sandbox's sandbox_id so it can reuse the underlying memory layout safely. Ids are process-local atomic counters and are not persisted to disk. Every call to Snapshot::from_file assigns a fresh id, and from_snapshot copies that id onto the resulting sandbox.

So sandboxes built from clones of the same in-memory Arc<Snapshot> are mutually restore-compatible, while sandboxes from independent from_file calls (even of the same path) are not.

performance (Linux/KVM)

End-to-end wall-clock from zero state to a completed Echo guest call.

Heap size	evolve	snapshot (check-hash)	snapshot (ignore-hash)	speedup
128 KB (default)	3.80 ms	2.73 ms	2.78 ms	1.4x
8 MB	8.89 ms	5.18 ms	2.96 ms	3.0x
64 MB	26.7 ms	19.9 ms	2.92 ms	9.1x
256 MB	88.3 ms	57.4 ms	3.33 ms	26.5x

known limitations

• core dumps from a from_snapshot sandbox lack binary_path and AT_ENTRY for Call snapshots, and mem_profile lacks accurate traces. The file format would need extending to fix these.
• max_guest_log_level is not plumbed through from_snapshot. It is also intrinsically ineffective for Call snapshots, this should be addressed orthogonally to this PR.
• guest-counter does not work on from_snapshot-built sandboxes.
• Mapped-region metadata is not preserved (region contents are absorbed into the memory blob, but snapshot.regions() is empty after load).
• to_file overwrites the target path non-atomically. A crash mid-write can leave a partially written file. Concurrent writers to the same path are not serialised.
• The file backing a loaded Snapshot (and any MultiUseSandbox built from it) must not be modified, truncated, renamed over, or deleted for the lifetime of the snapshot. On Linux this is unenforced. On Windows the OS refuses the operation with ERROR_USER_MAPPED_FILE (1224). (firecracker does this as well, see here

The memory file (pointed by backend_path when using File backend type, or pointed by mem_file_path) must be considered immutable from Firecracker and host point of view. It backs the guest OS memory for read access through the page cache. External modification to this file corrupts the guest memory and leads to undefined behavior.

future work

Typed error variants. Fuzz target for from_file. CoW overlay layers. Cross-hypervisor portability via sregs normalisation. Huge page support (MAP_HUGETLB). Atomic to_file via temp + rename + fsync. OCI distribution.

[Copilot]

Pull request overview

Adds first-class support for persisting Snapshots to disk and rehydrating sandboxes from those snapshots to avoid ELF parsing / guest init on cold start, using file-backed mappings for (near) zero-copy loads.

Changes:

• Implement Snapshot::to_file(), Snapshot::from_file(), and Snapshot::from_file_unchecked() with a versioned on-disk header + mmappable memory blob.
• Add MultiUseSandbox::from_snapshot() fast-path for sandbox creation directly from an in-memory or disk-loaded snapshot.
• Introduce file-backed ReadonlySharedMemory::from_file() and refactor SandboxMemoryLayout to expose enough stable layout fields for serialization.

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
src/hyperlight_host/src/sandbox/uninitialized_evolve.rs	Updates layout API usage (peb_address() accessor).
src/hyperlight_host/src/sandbox/uninitialized.rs	Ensures snapshot-based sandbox creation registers default HostPrint via FunctionRegistry::with_default_host_print().
src/hyperlight_host/src/sandbox/snapshot.rs	Defines snapshot file format + (de)serialization, file load/save APIs, sregs I/O, and extensive tests.
src/hyperlight_host/src/sandbox/initialized_multi_use.rs	Adds MultiUseSandbox::from_snapshot() instantiation path.
src/hyperlight_host/src/sandbox/host_funcs.rs	Adds FunctionRegistry::with_default_host_print() helper and makes default writer private.
src/hyperlight_host/src/mem/shared_mem.rs	Adds ReadonlySharedMemory::from_file() for file-backed snapshot memory mappings.
src/hyperlight_host/src/mem/mgr.rs	Updates layout field access for scratch I/O buffer sizes.
src/hyperlight_host/src/mem/memory_region.rs	Routes Snapshot regions through the Windows surrogate “ReadOnlyFile” mapping path.
src/hyperlight_host/src/mem/layout.rs	Refactors SandboxMemoryLayout to store key sizes directly and computes offsets via methods; updates PEB writing.
src/hyperlight_host/src/hypervisor/hyperlight_vm/x86_64.rs	Factors out apply_sregs() and updates peb_address() usage.
src/hyperlight_host/benches/benchmarks.rs	Adds benchmarks for snapshot file save/load and cold start via snapshot.
src/hyperlight_common/src/mem.rs	Adds write_to() helpers for GuestMemoryRegion and HyperlightPEB.
docs/snapshot-file-implementation-plan.md	Adds a detailed design/format plan and future work notes.

[andreiltd]

Looks good! I think the biggest issue, that have to be addressed here, is to NOT trust the parsed from binary header values and validate them before transforming it to the layout.

I would also recommend using winnow to improve mechanics and safety of parsing.