Add libplacebo GPU module with render and shader filters by ddennedy · Pull Request #1236 · mltframework/mlt

ddennedy · 2026-05-06T05:30:07Z

supersedes #1201

New module 'placebo' providing GPU-accelerated video processing via libplacebo. Includes two filters: - placebo.render: GPU scaling, debanding, dithering, and tonemapping with quality presets (fast/default/high_quality) - placebo.shader: Custom mpv-compatible .hook shader support Backend priority: D3D11 (Windows) -> Vulkan -> OpenGL. Vulkan loader is dynamically loaded on Windows when libplacebo is built without vk-proc-addr support. Features: - Singleton GPU context with thread-safe access - Shader cache persistence - Multiple scaling algorithms (ewa_lanczos, lanczos, mitchell, etc.) - Tone mapping (auto, clip, mobius, reinhard, hable, bt.2390, spline) - Graceful fallback to passthrough when no GPU is available The module is enabled by default but skipped automatically when libplacebo is not installed.

Use PRIu64/PRId64 from <inttypes.h> instead of %zu/%ld for size logging in the placebo module. Add libplacebo-dev packages to Ubuntu, Debian, and Fedora 42 CI workflows, and mingw-w64-x86_64-libplacebo to the MSYS2 MinGW64 workflow.

Break long mlt_log_info() call into multi-line format to match the project's clang-format rules (same style as load_cache above).

When multiple placebo filters are stacked on one clip, each filter previously did a full RAM→GPU upload and GPU→RAM download. The intermediate uploads are redundant because the next placebo filter would re-upload the same pixels immediately. Each filter now attaches its output texture to the mlt_frame via placebo_frame_put_tex(). The next placebo filter calls placebo_frame_take_tex() to grab it directly as source, skipping the upload. The download to RAM still happens every time (MLT expects the image buffer to be current for non-GPU filters). Staleness detection: put_tex records the RAM buffer pointer, take_tex compares it against the current pointer. If a CPU filter ran in between and requested a writable buffer (triggering a copy and new allocation), the pointers differ and take_tex returns NULL, falling back to a fresh upload. Also cleans up internal ticket-style comments (C1/W2/etc.) with descriptions of actual logic and pitfalls.

Add apply_shader_params() to override pl_hook DYNAMIC parameters from MLT animated properties (shader_param.* prefix) on every frame. Uses mlt_properties_anim_get_double/int to correctly resolve keyframe strings ("0=200;50=100") at the current frame position. Add base64 decoding for shader_text values prefixed with "base64:" to support inline shaders with characters that are problematic in MLT property strings.

Run clang-format-14 (matching CI) on filter_placebo_shader.c and gpu_context.c to fix designated initializer spacing, ternary line breaks, and long argument lists.

…into libplacebo

- Introduced new `mlt_image_private` format for module internal use. - Improved placebo texture reuse across filters.

ddennedy · 2026-05-06T20:42:35Z

@copilot resume review

- Remove incorrect pl_tex_destroy(src_tex) in error paths of the render and shader filters: src_tex is owned by the frame and freed automatically when placebo_frame_set_tex() replaces the "image" property, so manual destruction caused a double-free on frame close. - Add *image = NULL after failed placebo_frame_set_tex() in render and shader filters (consistent with filter_placebo_convert.c). - Guard filter_imageconvert's conversion_matrix access: mlt_image_private (and mlt_image_invalid) are out of range for the matrix indexed by [format - 1]; early-return 1 for any format outside [mlt_image_rgb, mlt_image_rgba64] to avoid an out-of-bounds read. Agent-Logs-Url: https://github.com/mltframework/mlt/sessions/f71fd929-5b62-4e71-864a-13291a3ec28e Co-authored-by: ddennedy <1146683+ddennedy@users.noreply.github.com>

Copilot · 2026-05-06T20:53:40Z

@copilot resume review

Resumed review and found three issues in the placebo module and related code, fixed in commit 739224f:

Double-free of src_tex (filter_placebo_render.c and filter_placebo_shader.c): when pl_tex_create failed for the destination texture, the code called pl_tex_destroy(gpu, &src_tex) — but src_tex is owned by the frame (registered via placebo_frame_set_tex upstream) and freed automatically when placebo_frame_set_tex replaces the "image" property. Manual destruction caused a double-free when the frame closed.
Dangling *image pointer (same files): when placebo_frame_set_tex failed after *image = dst_tex was set and dst_tex was manually freed, *image was left pointing to freed memory. Added *image = NULL (consistent with filter_placebo_convert.c).
Out-of-bounds array access (filter_imageconvert.c): conversion_matrix[*format - 1] is sized for formats mlt_image_rgb…mlt_image_rgba64 (indices 0–9). Now that mlt_image_private (12) can flow through the converter chain if the placebo converter fails, accessing index 11 was an out-of-bounds read. Added a bounds guard that returns 1 (not handled) for any format outside (mlt_image_none, mlt_image_invalid).

Copilot

Pull request overview

Copilot reviewed 32 out of 32 changed files in this pull request and generated 10 comments.

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

bmatherly

Thanks for this great PR. Here are some comments for your consideration. I did not run the code, so I apologize in advance if my comments are just lacking understanding.

bmatherly · 2026-05-07T02:51:12Z

+
+int placebo_frame_is_tex(mlt_frame frame, mlt_image_format format)
+{
+    const char *image_format = mlt_properties_get(MLT_FRAME_PROPERTIES(frame), "mlt_image_format");


I would suggest a different key for this property because "placebo" is, by definition, NOT a valid mlt_image_format. How about "_priv_format_id"? This could be a standard convention that we use for other modules that use the private type.

That was supposed to be "mlt_image_private" as described in #1235 (comment)
It is not really necessary to preface frame methods with underscore to hide them from serialization because they don't serialize, and here the value is possibly meaningful across modules.

changed to "mlt_image_private"

bmatherly · 2026-05-07T02:58:52Z

+void placebo_frame_set_requested_tex(mlt_frame frame, int requested)
+{
+    mlt_properties props = MLT_FRAME_PROPERTIES(frame);
+    int count = mlt_properties_get_int(props, "_placebo_requested");


Maybe we don't want to support or worry about this hypothetical future idea...

But, in a future world where avfilter also uses mlt_image_private, would this still work if we had a filter stack: placebo->avfilter->placebo?

I think the second placebo might try to interpret the private avframe type as a placebo. Maybe we could use a stack to push/pop private type requests?

It is not a hypothetical. It was needed now to make placebo->brightness(rgba)->placebo, placebo->sepia(yuv422)->placebo, and placebo->movit->placebo work, which is all I have tested so far (in addition to placebo alone and placebo->placebo). If avfilter follows this pattern, it can intermingle.

I think the second placebo might try to interpret the private avframe type as a placebo.

No, that is why it tags the requested mlt_format_private with property "_placebo_requested" and placebo.convert requires for that (placebo_frame_wants_tex()). It cannot trust mlt_image_format==mlt_image_private alone per my comment in the other PR linked above. The placebo render and shader filters also check property "mlt_image_private"=="placebo" after requesting it through mlt_frame_get_image() to be sure it can continue.

and mlt_properties_clear()

prepend the callback on the frame as-needed

and simplify cppcheck suppression for placebo module

ddennedy · 2026-05-08T19:45:43Z

I am still learning about libplacebo, and the new shader filter is obvious even though available shaders are rather limited to a few topics. However, I found the render filter rather odd. What exactly does it do for us? For example, it exposes scaling options but does no scaling. After some brief reading and a discussion with GPT:

The current placebo.render filter is best described as a GPU finishing pass, not a general-purpose scaling filter. Its clearest practical value today is debanding, dithering, and improving gradient quality before output, plus staying on the GPU when chained with placebo.shader.
In the current MLT integration, scaling is not meaningfully exercised. Both placebo.render and placebo.shader render to the incoming frame width and height, so the upscaler and downscaler settings are effectively dormant unless the filter is placed in a path where the requested frame size actually differs.
placebo.shader does not change the frame resolution in a way that would cause placebo.render to perform consumer-resolution scaling later. It also preserves the current frame dimensions.
Debanding and dithering are real, working features in this wrapper today. Those are the strongest reasons to use the filter in practice.
The current tone-mapping / color-mapping surface is much weaker than the deband/dither surface. Although pl_color_map_params and tone-mapping selection are wired, both source and destination frames are currently declared as sRGB, so the filter is not expressing a robust source-to-target color transform. In other words, the tone-mapping control exists, but the surrounding color-state information needed to make it fully meaningful is not really there yet.
Because of that, the metadata and user-facing description should emphasize “GPU finishing pass”, “debanding”, “dithering”, and “cleaner gradients before output”, rather than leading with scaling.
On future scope: if the goal is one simple finishing filter, debanding and dithering should stay in placebo.render. The most valuable next steps would be richer deband/dither controls and, separately, a proper color/HDR filter once source/target color metadata is propagated correctly.

A couple of my thoughts

Maybe this module should provide new normalizing filters. It can do color (de)linearization, image (pixel format and colorspace) conversion, deinterlace, and scaling. It is a little disconcerting that these new filters are hard-wired around 8-bit rgba.

Pixel formats
Libplacebo is not really built around one fixed FFmpeg-style “supported format list.” It is built around frames, planes, and GPU texture formats. In practice that means it can work with:

RGB and YUV/YCbCr sources
packed and planar layouts
single-plane and multi-plane images
chroma-subsampled formats like 4:4:4, 4:2:2, and 4:2:0
alpha
higher bit depths, not just 8-bit
integer and float textures

So for planning, 8-bit RGBA, 10-bit YUV420, 16-bit RGB, and HDR-oriented formats are all in scope conceptually.

The caveat is important: exact ingest/output depends on the upload path and the GPU/backend texture support. Libplacebo’s model is capability-based, not “these exact 27 formats always work everywhere.”

Color spaces / color metadata
Libplacebo is also designed to understand and transform color metadata, including:

RGB vs YCbCr representations
color matrices
transfer characteristics
color primaries
full vs limited range
SDR and HDR workflows
gamut mapping
tone mapping
linear-light processing where appropriate

So yes, converting to and from linear light is within libplacebo’s capability model. It also supports HDR-related workflows like PQ and HLG, and source/target color transforms rather than just “treat everything as sRGB.”

A practical mental model is:

Input can be something like 10-bit 4:2:0 BT.2020 PQ
Libplacebo can interpret that correctly
It can linearize internally when needed for scaling/color operations
It can tone-map / gamut-map to another target
Output can be something like BT.709 SDR, higher-bit-depth RGB, or another supported target frame layout

What libplacebo is good at
If you are planning a future integration, libplacebo is well-suited for:

high-bit-depth RGB workflows
YUV input without flattening everything early
HDR to SDR and HDR to HDR conversions
linear-light aware scaling / filtering
preserving color metadata instead of hardcoding sRGB
GPU-native processing with debanding, dithering, gamut mapping, tone mapping, and scaling in one pipeline

What not to assume
Do not assume:

every FFmpeg pixel format maps 1:1 with no adaptation
every backend supports every texture format equally
an integration automatically preserves YUV/high-bit-depth/linear support just because libplacebo can

That last point is where wrappers often narrow the capability.

The shortest accurate summary is:

Yes, libplacebo can handle linear-light aware color conversion.
Yes, it can work with higher-bit-depth RGB.
Yes, it can work with YUV pixel formats, including planar/chroma-subsampled video formats.
No, there is not one tiny fixed “supported pixel format” list independent of backend and upload path.

OK, clearly this PR is just a start, and shader is clearly general purpose and extensible. However, now I think this render filter should be refactored into 2 new filters with no scaling or tonemapping options: deband and dither.

D-Ogi and others added 9 commits February 1, 2026 22:48

Fix clang-format violation in save_cache log line

394d73c

Break long mlt_log_info() call into multi-line format to match the project's clang-format rules (same style as load_cache above).

Fix clang-format-14 violations in placebo module

cc02961

Run clang-format-14 (matching CI) on filter_placebo_shader.c and gpu_context.c to fix designated initializer spacing, ternary line breaks, and long argument lists.

Merge branch 'feature/placebo-module' of https://github.com/D-Ogi/mlt …

11bf94d

…into libplacebo

add MLT_PLACEBO_CACHE_PATH and replace atext()

3b112bb

Add placebo.convert filter

ce99364

- Introduced new `mlt_image_private` format for module internal use. - Improved placebo texture reuse across filters.

ddennedy changed the title ~~Libplacebo~~ Add libplacebo GPU module with render and shader filters May 6, 2026

ddennedy added this to the v7.40.0 milestone May 6, 2026

ddennedy added 2 commits May 6, 2026 09:54

fix cppcheck and redundant workflow runs

1def44b

fix ctest on msvc

e80a020

ddennedy marked this pull request as ready for review May 6, 2026 17:38

ddennedy requested a review from Copilot May 6, 2026 17:38

Copilot started reviewing on behalf of ddennedy May 6, 2026 17:39 View session

ddennedy added 2 commits May 6, 2026 11:03

docs for placebo private image handling

626cf8a

Enhance image conversion logic to handle private format cases

9f97a83

Copilot started work on behalf of ddennedy May 6, 2026 20:42 View session

ddennedy review requested due to automatic review settings May 6, 2026 20:53

ddennedy requested a review from Copilot May 6, 2026 20:53

Copilot started reviewing on behalf of ddennedy May 6, 2026 20:54 View session

Copilot finished work on behalf of ddennedy May 6, 2026 20:54

Copilot AI reviewed May 6, 2026

View reviewed changes

ddennedy and others added 3 commits May 6, 2026 14:16

Apply suggestions from code review

86df189

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

fix lazy allocation not thread-safe

7cd956a

fix races in placebo.shader

9a6e134

ddennedy added 2 commits May 6, 2026 14:47

Add permanent failure flag for GPU initialization and fix leak

f2a9e83

Add permanent failure and shutdown flags for GPU management

8d7260a

ddennedy mentioned this pull request May 6, 2026

Add libplacebo GPU module with render and shader filters #1201

Closed

bmatherly reviewed May 7, 2026

View reviewed changes

ddennedy added 8 commits May 6, 2026 21:49

use "mlt_image_private"

34b2717

and mlt_properties_clear()

add mlt_frame_prepend_convert_image()

ef2da1c

Merge branch 'refactor-image-convert' into libplacebo

50c1f24

remove placebo.convert filter

6c75bd1

prepend the callback on the frame as-needed

fix cppcheck

2dcf694

Merge branch 'refactor-image-convert' into libplacebo

148bfa2

update CMakeLists.txt to require libplacebo version 5.229

705b3e0

and simplify cppcheck suppression for placebo module

improve placebo.render meta & cleanup

4c74b79

Conversation

ddennedy commented May 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ddennedy commented May 6, 2026

Uh oh!

Copilot AI commented May 6, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bmatherly left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

bmatherly May 7, 2026

Choose a reason for hiding this comment

Uh oh!

ddennedy May 7, 2026

Choose a reason for hiding this comment

Uh oh!

ddennedy May 7, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

bmatherly May 7, 2026

Choose a reason for hiding this comment

Uh oh!

ddennedy May 7, 2026

Choose a reason for hiding this comment

Uh oh!

ddennedy commented May 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

A couple of my thoughts

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

ddennedy commented May 6, 2026 •

edited

Loading

ddennedy commented May 8, 2026 •

edited

Loading