feat: automatically fall back to VAE tiling when an untiled decode exceeds the backend buffer limit

[RapidMark]

On memory-constrained backends — integrated GPUs especially — a full-image VAE decode allocates a single compute buffer larger than the backend's maximum single-buffer/allocation size, and sd.cpp hard-fails instead of falling back to the tiling it already supports. The user has to know to pass --vae-tiling up front; otherwise the run crashes at the very end, after sampling has already completed.

Repro

AMD Radeon 8060S (Strix Halo, RDNA3.5 iGPU, 128 GB unified memory), Vulkan backend, Flux Krea-dev Q4 at 1024×1024, with no tiling flag:

[INFO ] stable-diffusion.cpp - sampling completed, taking 10.24s
ggml_vulkan: Requested buffer size exceeds device buffer size limit: ErrorOutOfDeviceMemory
[ERROR] ggml_extend.hpp - vae: failed to allocate the compute buffer
[ERROR] vae.hpp - vae decode compute failed
[ERROR] main.cpp - generate failed

The ~8.5 GB single-shot VAE decode buffer exceeds the iGPU's Vulkan per-buffer limit. The card has ample total memory (it shares 128 GB system RAM) — the failure is the per-buffer ceiling, not capacity. The whole gen is lost after a successful sampling pass.

Change

Add an automatic fallback to tiling, on by default, and keep it non-breaking:

• --vae-tiling stays exactly as it was — a boolean flag that forces tiling on.
• The auto-fallback is the new default. Before allocating the untiled decode buffer, the planned size is measured from the graph with ggml_gallocr_reserve_n_size (no-alloc planning, zero allocation) and compared against ggml_backend_buft_get_max_size(); if it won't fit, the decode goes straight to tiling. This is non-breaking — a decode that previously fit behaves identically, and one that previously OOM'd now recovers — and strictly safer. On CPU get_max_size() is SIZE_MAX, so it no-ops there.
• --no-vae-tiling-fallback disables the fallback for anyone who wants the old hard-fail behavior.
• A reactive backstop remains: if the untiled _compute still returns empty at runtime (e.g. the planned size fit the max but the device is genuinely full), it frees the buffer and retries once tiled — so a true OOM is also covered.

Implemented with a bool auto_tile appended to the end of sd_tiling_params_t (kept at the end so the C ABI stays backward-compatible; default true), the proactive probe in GGMLRunner::alloc_compute_buffer, and the fallback branch in VAE::decode.

Choosing the real graph-planned size (not a hardcoded bytes-per-pixel estimate) keeps it correct across every VAE architecture (SD/SDXL/Flux/Wan/LTX) and backend with no tuning.

Validation (AMD Radeon 8060S iGPU, Krea Q4, 1024²)

• default (no flag) → vae: untiled decode buffer exceeded the backend limit; retrying with tiling, completes, exit 0
• --no-vae-tiling-fallback → fails at decode, exit 1 (the old behavior, opt-in)
• --vae-tiling → tiles from the start, exit 0

The tiled GPU decode (~6.9 s) is also far faster than the usual workaround of routing the VAE to CPU (~29.5 s) to dodge the OOM, and is visually equivalent at 0.5 tile overlap.

Helps any constrained device, not just iGPUs — an 8 GB discrete card at high resolution hits the same per-buffer wall. Scoped to decode (where the failure occurs); encode has the same shape and could get the identical treatment later.

---

Thanks to @wbruna for pushing toward the proactive graph-planned size, and @stduhpf for catching that the original tristate would have broken the --vae-tiling syntax (this revision keeps it a plain flag + auto-by-default + opt-out).

[wbruna]

I chose reactive (retry on the real allocation failure) over proactive (estimate the buffer size and compare against ggml_backend_buft_get_max_size()) deliberately: a size estimate is VAE-architecture-specific (peak activation differs across SD/SDXL/Flux/Wan/LTX VAEs), so a hardcoded bytes-per-pixel constant would be brittle

Wouldn't be possible to check with the real value, calculated from the graph before the allocation?

[RapidMark]

Good call — done (pushed just now).

Instead of retrying on failure, the AUTO path now measures the planned compute-buffer size up front with ggml_gallocr_reserve_n_size (no-alloc planning, zero allocation) and compares it to ggml_backend_buft_get_max_size() before allocating; if it won't fit, it goes straight to tiling — no bytes-per-pixel estimate. On CPU get_max_size() is SIZE_MAX, so it no-ops there.

I kept the original retry-on-empty as a backstop for a genuine runtime OOM (planned size fits the max, but the device is actually full). Net effect on the auto path: the backend no longer prints its raw "allocation failed" error — just an INFO line and the tiled decode.

Validated on an AMD Radeon 8060S iGPU (Krea Q4, 1024²): --vae-tiling auto now logs only the INFO "retrying with tiling" and completes; off still fails; on still tiles.

[stduhpf]

I think having a fallback to vae tiling is a much welcome addition, but I'm having some small issues with the user experience there. Modifying the syntax of --vae-tiling arg from a flag to a tistate option will break previously working commands, and I think we could implement the same feature without breakling anything.

For example we could add a --vae-tiling-auto flag.

Alternatively, set "auto" tiling as default and add something like a --no-auto-vae-tiling to disable it.

[RapidMark]

Thanks — agreed, changing --vae-tiling's syntax wasn't worth the breakage. Pushed a revision (your alternative B):

• --vae-tiling is back to the original boolean flag (force tiling on).
• The auto-fallback is now the default — it only tiles when an untiled decode would exceed the backend's max buffer size, so it's non-breaking and strictly safer for existing commands (an untiled decode that previously fit behaves identically; one that previously OOM'd now recovers).
• --no-vae-tiling-fallback disables it for the old hard-fail behavior.

Validated on an AMD Radeon 8060S iGPU (Krea Q4, 1024²): default auto-recovers (logs vae: untiled decode buffer exceeded the backend limit; retrying with tiling, exit 0); --no-vae-tiling-fallback fails (exit 1); --vae-tiling tiles from the start (exit 0).