extension/llm/server: token-ID prompt segments for tool-use resume (V2b.1.5)

[mergennachin]

Warm resume (V2b.1) misses on agent loops: an assistant turn re-rendered from
its parsed tool call almost never re-tokenizes to the tokens the model actually
generated, so the resident state isn't an exact prefix and the worker resets.
On BFCL multi_turn the warm-resume hit rate was 0%.

Fix: carry the exact tokens instead of re-deriving them from text. The worker
returns generated_token_ids on done and accepts a prompt_segments form of
the prompt -- an ordered list of {"text"} chunks to tokenize and {"ids"} runs of
literal token ids (mutually exclusive with the plain prompt string); the
WorkerClient/SessionRuntime transport for that form was introduced with the
SessionRuntime boundary, and this commit makes the worker assemble and emit it.
The adapter-specific transcript glue lives in a new module, openai_transcript.py
(OpenAITranscriptState): it stores one record per assistant turn ({fingerprint,
ids}) and, on the next request, rebuilds the prompt as segments -- each prior
assistant turn is replaced with a unique sentinel, the conversation is rendered
once, and the rendered text is split on the sentinels with the stored ids spliced
back in. Tool results stay text (they re-tokenize deterministically). This logic
is the OpenAI adapter's concern, not the runtime's: SessionRuntime only sees a
PromptInput (text or segments).

Splicing is guarded so stale ids are never injected: a turn is substituted only
when the incoming assistant message fingerprint-matches the response we returned
(an edited or branched history, or a session reused for another conversation ->
text fallback; splicing stops at the first divergence and the now-stale tail is
pruned, and a regenerated turn is recorded at its position so it replaces the
stale record instead of shadowing later hits), and only when its ids faithfully
decode to what the client saw -- a stop-string trim kept post-stop tokens
resident but dropped them from the output, so the worker omits the ids and the
turn is re-rendered as text. Sentinel collisions / dropped sentinels also fall
back to text, and the worker's exact-token prefix check backstops the rest.

The context-window preflight counts what the worker actually assembles: for a
segment prompt it sums the literal {ids} run lengths and the tokenized {text}
chunks (not the rendered string), so a near-limit request agrees with the worker
rather than false-rejecting or failing mid-decode.

On BFCL multi_turn (per-conversation sessions) this moves the reuse fraction
from 0% to ~50% (exact_prefix hits where there were none); the single-turn AST
suite is unchanged (no prior assistant turn -> plain text prompt).

Review order: worker_loop.h (segment assembly + faithful generated_token_ids);
then the control plane (the new openai_transcript.py store + fingerprint-guarded
sentinel rendering, and the serving_chat wiring that builds the segments and
counts them for the context preflight); then tests and docs.

Part of #20001

[mergennachin]

Stack from ghstack (oldest at bottom):

• -> extension/llm/server: token-ID prompt segments for tool-use resume (V2b.1.5) #20161
• extension/llm/server: warm append-only session resume (V2b.1) #20160
• extension/llm/server: isolated multi-session serving (V2a) #20159
• Qwen3.5-MoE CUDA V2 foundation: one model, many isolated sessions #20117
• examples/models/qwen3_5_moe: CUDA Engine/Session adapter + OpenAI serving #20043
• extension/llm/server: document pi integration #19999
• extension/llm/server: worker-based OpenAI-compatible HTTP server #19994
• extension/llm/server: serving foundations (schemas, errors, templating, tools) #19993
• extension/llm/runner: Engine/Session C++ core + token-step primitives #19991

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/20161

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 4 New Failures, 1 Cancelled Job, 2 Unclassified Failures

As of commit a120263 with merge base eeb0646 ():

NEW FAILURES - The following jobs have failed:

• pull / unittest / linux / linux-job (gh)
  RuntimeError: Command docker exec -t 3fd5c072632a7ed640f8068a1bdd6c4452ced799e5534b55d618b40bf9e31508 /exec failed with exit code 1
• pull / unittest / macos / macos-job (gh)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• pull / unittest-editable / linux / linux-job (gh)
  RuntimeError: Command docker exec -t 1ce5288a847fb4f855ccc5df7dec145f534a812b70dd48af67cff3d19d2af347 /exec failed with exit code 1
• pull / unittest-editable / macos / macos-job (gh)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1

UNCLASSIFIED FAILURES - DrCI could not classify the following jobs because the workflow did not run on the merge base. The failures may be pre-existing on trunk or introduced by this PR:

• Build Windows Wheels / pytorch/executorch / build-wheel-py3_10-cpu (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  Process completed with exit code 1.
• Build Windows Wheels / pytorch/executorch / upload / upload-wheel-py3_10-cpu (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  Unable to download artifact(s): Artifact not found for name: pytorch_executorch__3.10_cpu_x64

CANCELLED JOB - The following job was cancelled. Please retry:

• Test Metal Backend / test-model-metal-e2e (nvidia, parakeet-tdt, quantized-int4-metal) / macos-job (gh)
  ##[error]The operation was canceled.

This comment was automatically generated by Dr. CI and updates every 15 minutes.