Optimizer: prefer no inlining in debug builds

[auduchinok]

inline functions are fully inlined on the call sites by F# compiler. This means the source information and actual calls are lost, so it's very hard to reliably debug code using such functions.

This PR prevents inlining inline functions in debug builds. This improves debugging experience by allowing setting breakpoints inside such functions and allowing stepping into them and checking the function arguments. It also allows IDEs to analyze the produced IL more reliably during debug.

The implementation handles two distinct cases. When possible, a normal call to the existing method is emitted. When a function contains SRTPs and no callable method is produced, a specialized method is created for each type instantiation.

An example inline function that can be called directly:

let inline add a =
    a + 1

[<EntryPoint>]
let main _ =
    let i = add 1
    0

The existing method is called:

An inline function with SRTPs:

let inline add a b =
    a + b

[<EntryPoint>]
let main _ =
    let i = add 1 2
    0

A specialized method is produced and called:

Open questions:

• Should we annotate specialized functions from other assemblies somehow? DebuggerNonUserCode?
• Should we whitelist something from FSharp.Core, like built-in operators?

Implements fsharp/fslang-suggestions#824 for debug builds.
Fixes #9555.

[github-actions]

❗ Release notes required

---

✅ Found changes and release notes in following paths:

Change path	Release notes path	Description
src/Compiler	docs/release-notes/.FSharp.Compiler.Service/11.0.100.md

[T-Gro]

re: Debugging

I would rely on people choosing to do "Just My Code" or not, in general.
For existing FSharp.Core functions, this might be indeed way too intrusive - are you thinking about handrolling the attributes, or a more general treatment when compiling fslib?

I would assume most of the "ugly to debug" fslib code uses statically optimized switches and is forced to be inlined even after this feature?

[auduchinok]

I would rely on people choosing to do "Just My Code" or not, in general.

Yes, IDE settings is going to be the primary way to control it. The problem is when an SRTP function from another assembly gets a specialized definition, the resulting method becomes a part of the user assembly. That's the reason I'm considering adding an attribute like DebuggerNonUserCode or another way to differentiate.

For existing FSharp.Core functions, this might be indeed way too intrusive - are you thinking about handrolling the attributes, or a more general treatment when compiling fslib?

I'm considering keeping inlining for functions in these modules:

• Microsoft.FSharp.Core.LanguagePrimitives.IntrinsicOperators
• Microsoft.FSharp.Core.Operators
• Microsoft.FSharp.Core.Operators.Checked
• Microsoft.FSharp.Core.Operators.OperatorIntrinsics
• Microsoft.FSharp.NativeInterop.NativePtr

That would allow to workaround the issue with NoDynamicInvocation and would also hide the most of the basic functions, like not or +, that the user is likely not going to step into anyway.

[auduchinok]

I've checked all NoDynamicInvocation usages on GitHub and the only ones that are problematic (i.e. hidden by the signature) are the ones in the FSharp.Core modules above, so I think it's safe enough to to whitelist these modules + check the attribute.

[T-Gro]

Ok, how do you want to whitelist those?
A module-level attribute ([<InlineEvenDebug>] like), or listing them in the compiler by name?

I agree with the selection, there is nothing surprising inside of those module's functions.

For the specialized SRTP definition, wouldn't [<CompilerGenerated>] work here?

[auduchinok]

Ok, how do you want to whitelist those?
A module-level attribute ([] like), or listing them in the compiler by name?

I'm going to try the second approach first, so it could work with older FSharp.Core too.

[auduchinok]

Finding 1: InlineIfLambda disabled in debug mode — user-defined CEs affected

It's done on purpose, since stepping through the code reliably seems more important in debug builds.

Finding 2: StateMachineHelpers module missing from force-inline list

It's done on purpose. If there're issues this can be changed in a future PR.

Finding 3: Specialization cache has unbounded growth potential

That's a good one. Is there some existing approach in the compiler that could be used here?

Finding 4: Deleted StateMachineTests test needs replacement

Thanks! This was deleted accidentally.

[T-Gro]

Review: Optimizer — prefer no inlining in debug builds

This is a well-structured PR with excellent test coverage (~40 new test cases). The goal of making inline functions debuggable is highly valuable. However, there are several issues that should be addressed before merge.

---

🔴 CRITICAL: State Machine Compilation Regression in Debug Builds

The PR disables InlineIfLambda inlining in debug mode (inlineIfLambda && cenv.settings.alwaysInline). This breaks state machine compilation for task { }, async { }, and all user-defined computation expressions in debug builds.

Evidence:

• Removed test: StateMachineTests.fs deletes "Debug-mode: mixing resumable and standard computation expressions compiles"
• tests/fsharp/tests.fs: asyncStackTraces and state-machines-non-optimized tests changed from --optimize- to --optimize+ — these were specifically testing non-optimized behavior
• TaskGeneratedCode baselines now show callvirt ... TaskBuilder::Run<int32> instead of the state machine MoveNext method

Root cause: Task/async builder methods (TaskBuilder.Run, TaskBuilder.Bind, etc.) are inline with InlineIfLambda parameters but:

• They're NOT in fslibForceInlineModules (only operators and intrinsics are)
• They don't have NoDynamicInvocation
• Therefore shouldForceInlineInDebug returns false
• Without inlining, if __useResumableCode then evaluates to false at runtime, falling through to the dynamic path

Impact: Every F# user's debug build would:

• Allocate closures and heap objects where state machines previously avoided them
• Produce different async stack traces
• Have degraded debug-mode performance for task { } and async { }

Recommendation: At minimum, shouldForceInlineInDebug should also force-inline vals whose return type involves ResumableCode<_,_> or recognize InlineIfLambda-bearing builder methods. Alternatively, add the FSharp.Core task builder modules to fslibForceInlineModules.

---

🟠 HIGH: ValInline.InlinedDefinition Uses the Zero Bit Pattern

InlinedDefinition maps to bit pattern 0b00 in ValFlags. Previously, 0b00 was unused (the InlineInfo getter would failwith "unreachable" for it). Zero is the default value for uninitialized data — if any code path creates ValFlags with default zero bits in those positions, the inline info will silently read as InlinedDefinition rather than being caught by an assertion.

While PickledBits correctly remaps it to Always before serialization, the zero-default concern remains for in-memory paths.

Recommendation: Use a different representation (e.g., a separate flag bit) or ensure the zero value remains unused.

---

🟠 HIGH: taccessPublic on Specialized Debug Vals

sharp let debugVal = Construct.NewVal(debugValName, m, None, debugValTy, Immutable, true, valReprInfo, taccessPublic, ...)

All specialized debug vals are created with taccessPublic regardless of the original function's accessibility. An internal inline function produces a public debug specialization. While these are local let bindings, IlxGen may emit them as public static methods.

Recommendation: Propagate the original val's accessibility: vref.Accessibility.

---

🟡 MEDIUM: No LanguageFeature Guard for Default-On Behavior Change

This changes default behavior for every Debug build (debug info on + optimizations off = standard dotnet build Debug). There's no gradual opt-in path — projects upgrading to this compiler silently get different debug-build behavior. Consider shipping behind a LanguageFeature flag (off-by-default in preview) initially, or at minimum require explicit MSBuild opt-in for the first release.

---

🟡 MEDIUM: CheckInlineValueIsComplete Disabled in Debug

When alwaysInline=false, incomplete inline values don't produce errors. A consumer compiling in release mode against a debug-compiled library might encounter errors that weren't reported when the library was built, creating confusing cross-compilation-mode failures.

---

🟡 MEDIUM: Closure Name Range Manipulation

The closure name collision fix creates synthetic ranges for cross-file copied expressions:
sharp Range.mkFileIndexRange eenv.cloc.Range.FileIndex expr.Range.Start expr.Range.End
This could confuse PDB/debugging (wrong file attribution). Consider using a counter that's keyed by the enclosing type instead.

---

✅ Positive Observations

1. Excellent test coverage with ~40 cases covering direct calls, SRTP specialization, cross-assembly, byrefs, accessibility, members, operators, recursive functions
2. Sound recursion prevention via dontInline map for both concrete and non-concrete type specializations
3. Correct byref handling — falls back when closure path can't handle byrefs
4. Cross-assembly design is correct — release consumers of debug-compiled libraries inline normally
5. Binary compatibility maintained via PickledBits remapping

---

Summary

The implementation is architecturally sound but needs to address the state machine regression before shipping. The removed/changed state machine tests should be restored, and the InlineIfLambda path needs to remain functional for resumable code patterns in debug builds.