(composefs backend): Linux 7.0.1 fails to boot with error "has no fs-verity digest"

[hecknt]

When attempting to boot a system off of the composefs backend, it fails to switch root. This does not happen on an equivalent system running 6.19.14.

Screenshots from a vm:

Kernel version used (taken from https://koji.fedoraproject.org/koji/buildinfo?buildID=2985771):

These errors are observed on the release version of bootc 1.15.1.

[Johan-Liebert1]

This seems like a case of verity being enforced or not. Could you share the kernel cmdline and the filesystem you're using? Also, did this happen after install to-disk or after install to-filesystem?

[hecknt]

This happens whether i run install to-disk or install to-filesystem. This same error also appears when upgrading from a (previously functional) system from kernel 6.19.14 up to 7.0.1.

In my example, it was installed via bootc install to-disk, and the root filesystem is btrfs. This is my kernel commandline:

rw composefs=d6869be26ac7cf657750069a1e6117e9a43f9df977a1b1e256360f2eac4da63eca91f2e4c557c9838a5393b0eeab20ccff8088e179011a86ca6907d39e09f3b9`

[marcoh00]

I'm seeing the same behavior with my Arch Linux image (where 7.0.2 left testing and hit the core repository today).

It uses an UKI created with (this did not change for the last >10 working images I built)

bootc container ukify --rootfs "/target" \
  --karg "rw" \
  -- \
  --signtool sbsign \
  --secureboot-private-key "/run/secrets/dbkey" \
  --secureboot-certificate "/run/secrets/dbcrt" \
  --measure \
  --json pretty \
  --output "/out/${kver}.efi"

I also tried to change the linux package to linux-lts (6.18), which works. I can also reproduce this using bootc install to-disk --loopback --filesystem {ext4,btrfs} and booting the image using qemu. So this seems to happen independently of the underlying filesystem.

Looks like it's either a kernel regression or a bug with how composefs generates the fs-verity information.

I'm a bit at a loss here on how to debug this. If I can help, please just say so. I'll try to make a minimal example later.

[marcoh00]

This Containerfile works (currently on 6.19.14):

FROM quay.io/fedora/fedora-bootc:44

RUN <<EORUN
set -euxo pipefail

dnf install -y systemd-boot-unsigned
EORUN

This one (7.0.3) will crash as seen above:

FROM quay.io/fedora/fedora-bootc:44

RUN <<EORUN
set -euxo pipefail

dnf install -y systemd-boot-unsigned 'dnf5-command(copr)'
dnf -y copr enable @kernel-vanilla/stable
dnf update -y
EORUN

And to test:

IMAGE="localhost/fedora-test-2174:broken"
FS_IMAGE=test.img
FILESYSTEM=btrfs
truncate -s 10G "$FS_IMAGE"
podman run \
    -e "RUST_LOG=debug" \
    -e "RUST_BACKTRACE=full" \
    --rm \
    --privileged \
    --pid=host \
    --security-opt label=type:unconfined_t \
    -v /dev:/dev  \
    -v /var/lib/containers:/var/lib/containers \
    -v .:/output \
    "$IMAGE" \
        bootc install to-disk \
        --filesystem "$FILESYSTEM" \
        --composefs-backend \
        --bootloader systemd \
        --wipe \
        --via-loopback "/output/${FS_IMAGE}"

and run via qemu (uefi firmware paths might differ - make a working copy of /usr/share/edk2/.../OVMF_VAR.4m.fd first):

qemu-system-x86_64 \
    -enable-kvm \
    -machine q35,accel=kvm \
    -cpu host \
    -m 2048 \
    -serial stdio \
    -drive "format=raw,file=${IMAGE},if=virtio" \
    -drive if=pflash,readonly=on,file=/usr/share/edk2/x64/OVMF_CODE.4m.fd \
    -drive if=pflash,file=OVMF_VARS.4m.fd

I'll have a look at the filesystems later.

This will probably also affect large parts of the CI and existing Fedora bootc installations when 7.0.x hits stable fedora eventually.

[cgwalters]

Thanks for the report, I'm looking at this.

[cgwalters]

I tossed an agent at this, I didn't deep verify yet but initial findings point at this being a regression from https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=f77f281b6118

Details

Following text is AI generated, only superficially human reviewed

Systems using overlayfs with verity=require (i.e. composefs) fail at boot
since Linux 7.0. The error, from ovl_validate_verity() in
fs/overlayfs/util.c:

lower file 'foo' has no fs-verity digest

This worked fine on 6.19.14. Upstream report:
#2174

Culprit:

f77f281b6118 ("fsverity: use a hashtable to find the fsverity_info")

Root cause:

ovl_ensure_verity_loaded() was written against the old contract of
fsverity_active(), which meant "is the verity info actually loaded into
memory?":

/* Before f77f281b6118 */
static inline bool fsverity_active(const struct inode *inode)
{
    return fsverity_get_info(inode) != NULL;
}

The overlayfs lazy-load guard relies on this distinction:

/* fs/overlayfs/util.c: ovl_ensure_verity_loaded() */
if (!fsverity_active(inode) && IS_VERITY(inode)) {
    /* info not loaded yet -- open the file to force it in */
    filp = kernel_file_open(datapath, O_RDONLY, current_cred());
    ...
}

After f77f281b6118, fsverity_active() returns true for any inode with
IS_VERITY set, regardless of whether the info is actually in the hash
table. The guard is therefore never entered, the load never happens, and
the subsequent fsverity_get_digest() finds nothing in the hash table and
returns 0, producing the error.

There is a second affected site in fs/overlayfs/copy_up.c:1042, where
fsverity_active() is used as a post-ovl_ensure_verity_loaded() check to
confirm the info is present. With the new semantics it always returns true
for verity inodes and no longer validates that the load actually succeeded.

Fix:

Replace fsverity_active() with fsverity_get_info() in
ovl_ensure_verity_loaded() and at the copy_up.c call site:

/* util.c */
if (IS_VERITY(inode) && !fsverity_get_info(inode)) {
    filp = kernel_file_open(datapath, O_RDONLY, current_cred());
    ...
}

/* copy_up.c */
if (WARN_ON_ONCE(lowerdata.dentry == NULL) ||
    ovl_ensure_verity_loaded(&lowerdata) ||
    !fsverity_get_info(d_inode(lowerdata.dentry))) {
        return false;
}

ovl_ensure_verity_loaded() was introduced in 0c71faf5a607 ("ovl: Handle
verity during copy-up", 2023) before the hashtable refactor and was not
updated to match the new semantics.

#regzbot introduced: f77f281b61183a5c0b87e6a4d101c70bd32c1c79

#regzbot monitor: #2174

[marcoh00]

It' s f77f281b61183a5c0b87e6a4d101c70bd32c1c79 for sure.

I compiled the kernel both at f77f281b61183a5c0b87e6a4d101c70bd32c1c79 and b0160e4501bb3572d9ef6e4f8edf758193ee37aa (its parent).

Former crashes as seen above, latter boots just fine.

[cgwalters]

https://lore.kernel.org/linux-fsdevel/bf0ba588-002b-4860-848e-e806b193840a@app.fastmail.com/T/#u