Add Nemotron-ASR streaming inference to Rust SDK

[rui-ren]

Add Nemotron-ASR streaming inference to Rust SDK"

Description

Ports the C# live audio transcription feature (PR #485) to the Rust SDK with full API parity.

The existing AudioClient only supports file-based transcription. This PR introduces LiveAudioTranscriptionSession that accepts continuous PCM audio chunks (e.g., from a microphone) and returns partial/final transcription results as an async stream.

What's included

New files

• sdk/rust/src/openai/live_audio_client.rs — Streaming session with start(), append(), get_transcription_stream(), stop(), plus types, cancellation support, and unit tests
• sdk/rust/tests/integration/live_audio_test.rs — E2E integration test with synthetic PCM audio
• samples/rust/live-audio-transcription-example/ — Full sample with real microphone capture (cpal) and resampling

Modified files

• sdk/rust/src/detail/core_interop.rs — Added StreamingRequestBuffer FFI struct and execute_command_with_binary() for binary audio data
• sdk/rust/src/openai/audio_client.rs — Added create_live_transcription_session() factory method
• sdk/rust/src/detail/model.rs, model_variant.rs — Wired factory method to Model
• sdk/rust/src/openai/mod.rs, src/lib.rs — Module registration and public exports
• sdk/rust/Cargo.toml — Added tokio-util dependency for CancellationToken

API surface

let audio_client = model.create_audio_client();
let session = audio_client.create_live_transcription_session();

session.settings.sample_rate = 16000;
session.settings.channels = 1;
session.settings.language = Some("en".into());

session.start(None).await?;

// Push audio from microphone callback
session.append(&pcm_bytes, None).await?;

// Read results as async stream
use tokio_stream::StreamExt;
let mut stream = session.get_transcription_stream()?;
while let Some(result) = stream.next().await {
    let result = result?;
    println!("{}", result.content[0].text);
}

session.stop(None).await?;

C# API parity

C#	Rust	Status
CreateLiveTranscriptionSession()	create_live_transcription_session()	✅
StartAsync(CancellationToken)	start(Option<CancellationToken>)	✅
AppendAsync(ReadOnlyMemory<byte>, CancellationToken)	append(&[u8], Option<CancellationToken>)	✅
GetTranscriptionStream(CancellationToken)	get_transcription_stream()	✅
StopAsync(CancellationToken) + cancel-safe cleanup	stop(Option<CancellationToken>) + cancel-safe cleanup	✅
IAsyncDisposable.DisposeAsync()	Drop with best-effort native stop	✅
LiveAudioTranscriptionResponse.Content[0].Text	response.content[0].text	✅
LiveAudioTranscriptionResponse.Content[0].Transcript	response.content[0].transcript	✅
LiveAudioTranscriptionResponse.IsFinal	response.is_final	✅
LiveAudioTranscriptionResponse.StartTime/EndTime	response.start_time / response.end_time	✅
LiveAudioTranscriptionOptions (SampleRate, Channels, BitsPerSample, Language, PushQueueCapacity)	LiveAudioTranscriptionOptions (sample_rate, channels, bits_per_sample, language, push_queue_capacity)	✅
CoreErrorResponse.TryParse()	CoreErrorResponse::try_parse()	✅
Native commands: audio_stream_start, audio_stream_push, audio_stream_stop	Same commands via execute_command / execute_command_with_binary	✅

Design highlights

• CancellationToken support — start/append/stop accept Option<CancellationToken> via tokio_util::sync::CancellationToken
• Cancel-safe stop — stop() always performs native audio_stream_stop even if token fires, preventing native session leaks (matches C# StopAsync pattern)
• Response envelope — LiveAudioTranscriptionResponse uses content: Vec<ContentPart> matching C#'s ConversationItem.Content[0].Text/Transcript
• Bounded push queue — Backpressure via bounded channel (capacity=100); prevents unbounded memory growth
• Push loop on blocking thread — execute_command_with_binary FFI calls run on spawn_blocking, keeping async runtime free
• Settings freeze — Audio format settings are cloned at start() and immutable during the session
• Drop safety — Best-effort synchronous audio_stream_stop in Drop to prevent native session leaks
• FFI null pointer safety — Empty binary slices use std::ptr::null() to avoid dangling pointer across FFI boundary

Verified working

• ✅ SDK build succeeds (0 errors, 0 clippy warnings)
• ✅ 13 unit tests passing (JSON deserialization, settings defaults, error parsing, content envelope)
• ✅ E2E pipeline: Microphone (48kHz/2ch/F32) → Resample (16kHz/mono/16-bit) → SDK → Core.dll → onnxruntime-genai.dll → nemotron model
• ✅ Synthetic audio test: 30 chunks (96KB PCM) pushed with clean session lifecycle
• ✅ Live microphone test: real-time capture, session start/stop, no native errors

Stats

• 14 files changed, 1,329 additions, 2 deletions

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
foundry-local	Ready	Preview, Comment	Apr 24, 2026 9:12pm

[Copilot]

Pull request overview

This PR adds live (chunked) PCM audio transcription streaming to the Foundry Local Rust SDK, aligning the Rust API with the existing C# live audio transcription session feature and extending the SDK beyond file-based transcription.

Changes:

• Introduces LiveAudioTranscriptionSession + associated response/options/types and stream wrapper in the Rust SDK.
• Extends the Rust FFI bridge with execute_command_with_binary() to send JSON params + binary PCM payloads.
• Adds integration test coverage and a new Rust sample demonstrating microphone capture (cpal) and streaming transcription.

Reviewed changes

Copilot reviewed 14 out of 14 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
sdk/rust/src/openai/live_audio_client.rs	New streaming session implementation, types, cancellation support, and unit tests
sdk/rust/src/detail/core_interop.rs	Adds StreamingRequestBuffer + optional execute_command_with_binary symbol support
sdk/rust/src/openai/audio_client.rs	Adds create_live_transcription_session() factory method
sdk/rust/src/detail/model.rs	Exposes Model::create_live_transcription_session()
sdk/rust/src/detail/model_variant.rs	Wires variant factory for live transcription sessions
sdk/rust/src/openai/mod.rs	Registers and re-exports live audio transcription module/types
sdk/rust/src/lib.rs	Public re-exports for the new live transcription session/types
sdk/rust/Cargo.toml	Adds tokio-util for CancellationToken
sdk/rust/tests/integration/main.rs	Registers the new integration test module
sdk/rust/tests/integration/live_audio_test.rs	New E2E-ish integration test using synthetic PCM audio
samples/rust/live-audio-transcription-example/src/main.rs	New microphone/synthetic streaming transcription sample
samples/rust/live-audio-transcription-example/Cargo.toml	Declares sample dependencies (cpal, tokio, sdk path dep)
samples/rust/Cargo.toml	Adds the new sample crate to the workspace
codex-feedback.md	Adds review/validation notes for the feature

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[bmehta001]

Will this leak the handle if we cancel? Should cancellation call audio_stream_stop?

[rui-ren]

Fix the leak concerns. Thanks

[nenad1002]

Just clone

[nenad1002]

Run cargo fmt & cargo clippy

[kunal-vaishnavi]

Same comment as here

[Copilot]

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

On a fatal push error, the push loop breaks but the output channel sender stored in session state remains alive, so get_transcription_stream() will not terminate (it will yield the error once, then await forever) unless stop()/Drop is called to drop output_tx. Consider mirroring the JS/C# behavior by ensuring the stream is completed on fatal push errors (e.g., drop/close the state-held sender, or send a control event to an async task that finalizes state and closes the output channel).

Suggested change

	break;
	// Fatal push failures are terminal for the transcription stream.
	// Drop the loop-owned sender and return immediately so the stream
	// can complete once no other senders remain.
	drop(output_tx);
	return;

[Copilot]

This integration test swallows transcription stream errors: the spawned read task only logs Err(e) and breaks, but the test still passes (especially if results ends up empty, since the final assertions are inside a for loop). To make this test meaningful as an E2E signal, propagate the first stream error back to the test (e.g., store it in shared state and assert!(error.is_none()) after stop()), and/or assert that at least one valid response was received once the session completes.

[Copilot]

CoreErrorResponse::try_parse(&e.to_string()) will never successfully parse JSON from the native core because FoundryLocalError’s Display format prefixes the message (e.g., "command execution error: ..."). This makes code always fall back to "UNKNOWN" even when the core returns structured JSON. Match on FoundryLocalError::CommandExecution { reason } (or otherwise access the raw reason string) and pass that to try_parse instead of e.to_string().

Suggested change

	let code = CoreErrorResponse::try_parse(&e.to_string())
	.map(|ei| ei.code)
	.unwrap_or_else(|| "UNKNOWN".into());
	let code = match &e {
	FoundryLocalError::CommandExecution { reason } => {
	CoreErrorResponse::try_parse(reason)
	.map(|ei| ei.code)
	.unwrap_or_else(|| "UNKNOWN".into())
	}
	_ => "UNKNOWN".into(),
	};