Skip to content

Immutable OS Paradigms

Warmwind's per-agent containers boot a read-only rootfs with an overlay for runtime writes. This article covers the building blocks: OSTree for versioning, dm-verity for integrity, A/B updates, SquashFS + OverlayFS layering, minimal rootfs construction, and the complete boot chain for a production kiosk OS.

What "Immutable" Means

A read-only root filesystem where runtime state lives in an overlay or tmpfs. Benefits: atomic updates, guaranteed rollback, tamper detection, reproducibility.

graph LR
    Lower["SquashFS rootfs"] --> Merged["Merged View"]
    Upper["OverlayFS upper"] --> Merged

Three Viable Approaches to Immutable OS

There is no single "right" answer. Three approaches have emerged, each with different tradeoffs:

Approach Model Strengths Tradeoffs
bootc + OSTree Container-native: build OS images as OCI containers, deploy via OSTree Familiar Containerfile workflow, OCI registries for distribution, atomic rollback Needs OSTree infrastructure, Fedora/RHEL ecosystem focus
systemd-sysupdate Image-based: fetch versioned filesystem images, A/B swap Simple model, distro-agnostic, GNOME OS moving to this Newer, less ecosystem tooling, no content deduplication
Nix / NixOS Functional: entire OS declared in Nix expressions, content-addressed store Fully reproducible, rollback via generations, hermetic builds Steep learning curve, own package ecosystem, large store

bootc + OSTree (container-native path)

bootc lets you define an OS image using a standard Containerfile, push it to an OCI registry, and deploy it to bare metal or VMs via OSTree.

# Example: Warmwind kiosk image as a Containerfile
FROM quay.io/fedora/fedora-bootc:40
RUN dnf install -y sway wayvnc chromium && dnf clean all
COPY warmwind.conf /etc/sway/config.d/
RUN systemctl enable sway.service

Your OCI expertise maps directly

bootc lets you build immutable OS images using the same Containerfile/Dockerfile format and OCI registries you already use. Your bash framework that produces OS images? That's literally what Warmwind needs for their container build pipeline.

OSTree fundamentals

OSTree stores complete filesystem trees as content-addressed objects (like Git). Each deployment is an atomic symlink swap.

# Initialize a repository
ostree init --repo=/ostree/repo --mode=archive

# Commit a rootfs directory
ostree commit --repo=/ostree/repo \
    --branch=warmwind/stable \
    --subject="v2.1.0 - updated Chromium" \
    /path/to/rootfs

# Deploy (atomic -- symlink swap)
ostree admin deploy warmwind/stable

# Rollback to previous deployment
ostree admin undeploy 1
Concept Git equivalent
Commit Commit (content-addressed tree)
Branch Branch (ref pointing to commit)
Deploy Checkout (atomic symlink to /ostree/deploy/)
Rollback Reset to previous commit

OSTree deduplicates files via hardlinks. Two deployments sharing 90% of files use only 10% extra disk.

systemd-sysupdate

GNOME OS is moving to systemd-sysupdate for image-based updates. It fetches versioned partition or filesystem images and swaps them atomically.

# /etc/sysupdate.d/rootfs.conf
[Source]
Type=url-file
Path=https://images.warmwind.space/
MatchPattern=rootfs_@v.img.zst

[Target]
Type=partition
Path=/dev/disk/by-partlabel/rootfs-*
MatchPattern=rootfs-@v

Nix (functional, fully reproducible)

Nix builds the entire OS from a declarative specification. Every package is stored at /nix/store/<hash>-<name>/, making builds fully reproducible and rollbacks trivial (switch symlink to a previous generation).

# NixOS module for a kiosk
{ config, pkgs, ... }: {
  services.cage = {
    enable = true;
    program = "${pkgs.chromium}/bin/chromium --kiosk https://agent.local";
  };
  environment.systemPackages = [ pkgs.wayvnc ];
}

dm-verity: Cryptographic Root Verification

dm-verity creates a device-mapper target that verifies every block read against a Merkle tree hash:

# Create verity hash tree
veritysetup format /dev/sda2 /dev/sda3
# Output: Root hash: a1b2c3d4...

# Activate verified device
veritysetup open /dev/sda2 verified-root /dev/sda3 a1b2c3d4...

# Mount the verified device
mount -o ro /dev/mapper/verified-root /mnt

Any tampered block returns I/O error. The root hash is stored in a signed boot partition or kernel command line (verified by Secure Boot).

A/B Partition Updates

graph LR
    subgraph "Partition Layout"
        BOOT["EFI System<br/>Partition"]
        A["Slot A (active)<br/>rootfs v2.0"]
        B["Slot B (standby)<br/>rootfs v2.1"]
        DATA["Data partition<br/>persistent state"]
    end
    BOOT --> A
    BOOT -. "next boot" .-> B

RAUC and SWUpdate manage A/B updates:

# RAUC system.conf
[system]
compatible=warmwind-kiosk
bootloader=efi

[slot.rootfs.0]
device=/dev/sda2
type=ext4
bootname=A

[slot.rootfs.1]
device=/dev/sda3
type=ext4
bootname=B

Update flow: write new image to inactive slot, verify hash, mark active on next boot. Failed boot? Watchdog triggers rollback to previous slot.

Building Minimal Debian Rootfs

# mmdebstrap: faster, more reproducible than debootstrap
mmdebstrap --variant=minbase \
    --include=systemd,dbus,sway,wayvnc,chromium \
    --customize-hook='echo "warmwind" > "$1/etc/hostname"' \
    --customize-hook='systemctl --root="$1" enable sway.service' \
    bookworm rootfs/ http://deb.debian.org/debian

# Result: ~300MB rootfs with full kiosk stack

# Create SquashFS image
mksquashfs rootfs/ rootfs.squashfs -comp zstd -Xcompression-level 19

mmdebstrap is preferred over debootstrap for reproducibility: it resolves dependencies in a single pass and supports hooks for customization.

SquashFS + OverlayFS

# Mount read-only SquashFS
mount -t squashfs -o ro rootfs.squashfs /lower

# Create writable overlay
mkdir -p /upper /work /merged
mount -t overlay overlay \
    -o lowerdir=/lower,upperdir=/upper,workdir=/work \
    /merged

# Now /merged shows the SquashFS contents but writes go to /upper

OverlayFS semantics: reads check upper first, then lower. Writes always go to upper. Deletes create "whiteout" entries in upper.

Reproducible Builds

Goal: same source + same build environment = bit-identical output.

Approach Mechanism
Nix Content-addressed store, hermetic builds, /nix/store/hash-name/
Debian reproducible SOURCE_DATE_EPOCH, strip-nondeterminism, diffoscope for comparison
Container builds Fixed base image hash, pinned package versions, --no-install-recommends 
# Nix: build a derivation reproducibly
nix-build -A warmwind-rootfs --check  # rebuilds and compares hashes

# Debian: set epoch for reproducible timestamps
export SOURCE_DATE_EPOCH=$(date -d "2025-01-01" +%s)
mmdebstrap --variant=minbase bookworm rootfs/

Complete Boot Chain (Warmwind Production)

graph LR
    UEFI["UEFI"] --> Boot["systemd-boot"] --> Initrd["initramfs"] --> Verity["dm-verity"] --> Overlay["OverlayFS"] --> Systemd["systemd"]
    Systemd --> Sway["Sway"] --> VNC["WayVNC"]
    Sway --> Chrome["Chromium"]

Step by step:

  1. UEFI Secure Boot verifies the signed bootloader (systemd-boot).
  2. systemd-boot loads kernel + initramfs (both signed).
  3. initramfs runs veritysetup open with root hash from kernel cmdline.
  4. dm-verity maps the SquashFS block device with integrity checking.
  5. OverlayFS mounts verified SquashFS as lower, tmpfs as upper.
  6. systemd boots with read-only root; /etc overlay for runtime config.
  7. Sway starts on headless backend; WayVNC attaches for remote access.
  8. Chromium launches in kiosk mode, connecting to agent backend.

Every boot is identical. Writes (logs, temp files) vanish on reboot.

What's new (2025--2026)

bootc gaining traction as the Docker-native path to immutable OS -- build with Containerfile, push to OCI registry, deploy to bare metal.

Lennart Poettering left Microsoft in January 2026 and co-founded Amutable with kernel VFS maintainer Christian Brauner.

systemd-repart + systemd-sysext are maturing as image composition tools, enabling modular OS images assembled from signed extensions.

Glossary

OSTree
Content-addressed filesystem versioning system. Atomic deploys via symlink swap.
dm-verity
Device-mapper target verifying block integrity via Merkle tree. Read-only.
SquashFS
Compressed read-only filesystem. High compression ratio, random access support.
OverlayFS
Union filesystem layering a writable upper on a read-only lower. In mainline Linux since 3.18.
A/B update
Dual-partition scheme: update the inactive slot, swap on reboot, rollback if boot fails.
RAUC
Robust Auto-Update Controller. Manages A/B updates with signature verification.
mmdebstrap
Reproducible Debian rootfs builder. Single-pass dependency resolution, hook system.
Nix
Purely functional package manager. Content-addressed store ensures reproducibility.
Whiteout
OverlayFS marker indicating a file in the lower layer has been deleted in the upper.