Fixed window crashes on macOS

[tomoyanonymous]

This was discussed on Dicord before, that the open_parented example crashes only on macOS build.
I also found that some gui examples in nih-plug crashes similarly because of null pointer dereference.

I finally found these issues are different problem but both are now fixed.

The problem 1:The open_parented example was crashing with a null pointer dereference when libraries like softbuffer attempted to create rendering surfaces. The crash occurred because child windows were not properly storing a reference to their parent window.

When creating a parented (child) window, the WindowInner::ns_window field was set to None.
This caused issues when external libraries called raw_window_handle().
raw_window_handle() would return null for the ns_window field.
Libraries like softbuffer would receive a null window handle and crash when trying to create rendering contexts
Attempting to get the window from the view via [ns_view window] also returned nil because the view's window property is not immediately updated after addSubview is called—it only gets set after the viewWillMoveToWindow or viewDidMoveToWindow callbacks are invoked.

So open_parented() was modified to extract the parent window from the parent's RawWindowHandle and store it in the child's WindowInner.

The Problem 2: Applications using wgpu or other Metal-based rendering libraries were crashing when attempting to create Metal surfaces. The crash occurred in wgpu's Metal backend with the null pointer dereference.

The custom NSView created by baseview was not layer-backed. When wgpu's Metal backend tried to obtain the view's layer to create a Metal surface, it received nil.

create_view() in view.rs was modified to make views layer-backed and attach a CAMetalLayer.
set_frame_size hook is not necessary for fixing crash but it will be needed for window resizing.

to test:
Run the open_parented example on debug build; confirm no softbuffer null-pointer panic.
Run a nih-plug/examples/gain_gui_iced example on debug build.

[micahrj]

Currently, the ns_window field on WindowInner is only set in the non-parented case, when baseview is responsible for creating and managing the NSWindow, and this is very much intentional:

baseview/src/macos/window.rs

Lines 61 to 62 in 237d323

	/// Only set if we created the parent window, i.e. we are running in
	/// parentless mode

In the parented case, the host is responsible for managing the NSWindow, and baseview should not be performing operations like resizing or closing it. This PR will cause baseview to incorrectly resize the parent NSWindow in Window::resize:

baseview/src/macos/window.rs

Lines 330 to 333 in 237d323

	// If this is a standalone window then we'll also need to resize the window itself
	if let Some(ns_window) = self.inner.ns_window.get() {
	unsafe { NSWindow::setContentSize_(ns_window, size) };
	}

and to incorrectly close it in Window::close:

baseview/src/macos/window.rs

Lines 92 to 95 in 237d323

	// Close the window if in non-parented mode
	if let Some(ns_window) = self.ns_window.take() {
	ns_window.close();
	}

[micahrj]

The problem 1:The open_parented example was crashing with a null pointer dereference when libraries like softbuffer attempted to create rendering surfaces. The crash occurred because child windows were not properly storing a reference to their parent window.

When creating a parented (child) window, the WindowInner::ns_window field was set to None. This caused issues when external libraries called raw_window_handle(). raw_window_handle() would return null for the ns_window field. Libraries like softbuffer would receive a null window handle and crash when trying to create rendering contexts

This should not occur with the latest version of softbuffer, as it uses raw-window-handle 0.6, where the AppKitWindowHandle struct doesn't even have an ns_window field: https://docs.rs/raw-window-handle/0.6.2/raw_window_handle/struct.AppKitWindowHandle.html

[dathinaios]

We had a discussion on discord some time ago regarding this PR. To quote myself and replies:

Ok so I have been chasing a bug and it is related to the fork.

In baseview the change made in the tomoyanonymous fork was causing Reaper to crash when adding a second instance of my plugin in the same FX chain window and the first editor handle got dropped. Projects that already had two instances would just get stuck crashing without being able to do much.

The issue was that parented macOS windows started storing the host parent NSWindow in WindowInner.ns_window, and later close() treated that as an owned standalone window.

https://github.com/tomoyanonymous/baseview/blob/macos_fix/src/macos/window.rs#L152-L168

And the fix was to restore the old parented-window behavior and keep ns_window as None there, so drop/close does not try to close the host window:

https://github.com/dathinaios/baseview/blob/eef928b7340dfb579ee6c6a66a79f98c7dd68bd7/src/macos/window.rs#L152-L160

@micahrj had a look and confirmed that the change in this PR is not in a good state:

all of the stuff about the NSWindow and softbuffer is just completely wrong and unnecessary

and it was confirmed through further discussion. You can read here.

So I believe we can close this PR.

[prokopyl]

So I believe we can close this PR.

I was gonna come to the same conclusion but you beat me to it 😄

I completely agree with @micahrj here, overriding the ns_view field here is definitely wrong, as it is meant only for the case where baseview is the owner of that window. Doing so will definitely wreck havoc on the host's window (as evidenced by the findings above).

The issue of the open_parented example not even working on macOS is still an issue though, and I have made #238 to address this in a more sensitive manner. (@dathinaios if you have the time to check if that patch does not have the same issue, please do, that would be very appreciated ^^)

I could not find a way to easily reproduce the Metal Layer issue (the nih_plug example relies on a completely outdated version of baseview), but it seems pretty crazy to me to add a CAMetalLayer on every created window, regardless of whether it uses Metal, OpenGL, or software rendering.

So for those reasons, I will close this PR, and address those two issues separately.