FileStream.Unix: prevent open to cause setting the controlling terminal.

[tmds]

This passes the O_NOCTTY flag to prevent terminal devices from becoming the controlling terminal of the process.
Without this flag, opening a terminal device (like a serial port) on a process with no controlling terminal would make it the controlling terminal. When the serial port then disconnects, the .NET process can get an unexpected SIGHUP which causes it to terminate.

Fixes #129156.

@adamsitnik ptal.

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/area-system-io
See info in area-owners.md if you want to be subscribed.

[Copilot]

Pull request overview

Updates the Unix SystemNative_Open shim in System.Native to always include O_NOCTTY, preventing terminal/TTY device nodes opened via File/FileStream (and other Interop.Sys.Open call sites) from implicitly becoming the process controlling terminal.

Changes:

• Adds O_NOCTTY to the open(2) flags used by SystemNative_Open on Unix.
• Documents the intent inline with a short comment.

Comments suppressed due to low confidence (1)

src/native/libs/System.Native/pal_io.c:356

• On platforms without O_CLOEXEC support, we always call fcntl(result, F_SETFD, FD_CLOEXEC) when PAL_O_CLOEXEC was requested, even if open() failed (result < 0). That can overwrite errno from the failing open() (e.g., ENOENT -> EBADF) and cause managed code to observe the wrong error. It also doesn't retry fcntl on EINTR or handle a rare fcntl failure by closing the fd.

    while ((result = open(path, flags, (mode_t)mode)) < 0 && errno == EINTR);
#if !HAVE_O_CLOEXEC
    if (old_flags & PAL_O_CLOEXEC)
    {
        fcntl(result, F_SETFD, FD_CLOEXEC);
    }

[adamsitnik]

@tmds It LGTM, but would you be able to provide a regression test? With ProcessStartInfo.StartDetached you can start a process that calls setsid after fork:

runtime/src/native/libs/System.Native/pal_process.c

Lines 870 to 872 in 4b6b5d0

	// Start the child in a new session when startDetached is set, making it independent
	// of the parent's process group and terminal.
	if (startDetached && setsid() == -1)

Thanks!

[tmds]

It LGTM, but would you be able to provide a regression test? With ProcessStartInfo.StartDetached you can start a process that calls setsid after fork:

I haven't included a test because it would need some interop to create a pty the child could open. I'm not sure the complexity of the test justifies how likely this is to regress.

[adamsitnik]

It LGTM, but would you be able to provide a regression test? With ProcessStartInfo.StartDetached you can start a process that calls setsid after fork:

I haven't included a test because it would need some interop to create a pty the child could open. I'm not sure the complexity of the test justifies how likely this is to regress.

Sounds reasonable to me. Let's revisit it once we implement #128565

[jkotas]

What are the chances that some app depends on this behavior and be broken by this change?

[tmds]

What are the chances that some app depends on this behavior and be broken by this change?

Very low. It would be weird for a process that has no controlling terminal to try and set one.

[tmds]

Very low. It would be weird for a process that has no controlling terminal to try and set one.

The main use-case where you want to set the controlling terminal is pseudo terminals (#128565). Something like https://github.com/mitchdenny/hex1b does it through interop. That library is calling forkpty.

As a side-effect of open, it is more likely to be a bug than it is used as a feature.