fix(ai-openai): migrate WebRTC realtime adapter to OpenAI GA API#699
Merged
tombeckenham merged 5 commits intoJun 10, 2026
Merged
Conversation
Contributor
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (2)
🚧 Files skipped from review as they are similar to previous changes (2)
📝 WalkthroughWalkthroughMigrate the OpenAI realtime adapter from Beta to GA: mint ephemeral tokens via POST /v1/realtime/client_secrets, POST SDP to /v1/realtime/calls, construct GA-shaped session.update payloads, rename GA event/content keys, and update types, tests, docs, and examples. ChangesOpenAI Realtime Adapter Fixes
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Completes the GA migration started in this PR so the whole realtime flow works against OpenAI's GA API (the Beta shape was shut down 2026-05-12): - openaiRealtimeToken() mints ephemeral keys via POST /v1/realtime/client_secrets (the Beta /v1/realtime/sessions endpoint is retired) and parses the GA top-level value/expires_at response shape - session.update payloads use the GA shape via a new pure buildSessionUpdate() helper: required session.type, audio.input.*, audio.output.voice, output_modalities, max_output_tokens; temperature (removed in GA) is dropped with a debug log instead of getting the whole update rejected with unknown_parameter - server events handled under GA names (response.output_audio_transcript.*, response.output_audio.*, output_text/output_audio content parts) - removed the now-unused model local in createWebRTCConnection (the GA /calls endpoint rejects ?model=; the model is bound to the ephemeral key) - default model gpt-realtime; dead gpt-4o-(mini-)realtime-preview ids (shut down 2026-05-07) removed from OpenAIRealtimeModel, docs, and examples - unit tests for the session.update payload and client-secret request/response shapes; changeset added Live-verified against the OpenAI API: client_secrets 200 (ek_ token), /v1/realtime/calls 201 with SDP answer, and session.updated echoing voice, semantic VAD, tools, output_modalities, and max_output_tokens. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
|
Thanks @amitsaroj — the two endpoint fixes here were exactly right, but on review the GA migration needed a few more pieces to actually work end to end, so I've pushed a completing commit (676dd84) to this branch (maintainer edit):
Live-verified against OpenAI's GA API: token minting (200, |
|
View your CI Pipeline Execution ↗ for commit 676dd84
☁️ Nx Cloud last updated this comment at |
@tanstack/ai
@tanstack/ai-anthropic
@tanstack/ai-client
@tanstack/ai-code-mode
@tanstack/ai-code-mode-skills
@tanstack/ai-devtools-core
@tanstack/ai-elevenlabs
@tanstack/ai-event-client
@tanstack/ai-fal
@tanstack/ai-gemini
@tanstack/ai-grok
@tanstack/ai-groq
@tanstack/ai-isolate-cloudflare
@tanstack/ai-isolate-node
@tanstack/ai-isolate-quickjs
@tanstack/ai-mcp
@tanstack/ai-ollama
@tanstack/ai-openai
@tanstack/ai-openrouter
@tanstack/ai-preact
@tanstack/ai-react
@tanstack/ai-react-ui
@tanstack/ai-solid
@tanstack/ai-solid-ui
@tanstack/ai-svelte
@tanstack/ai-utils
@tanstack/ai-vue
@tanstack/ai-vue-ui
@tanstack/openai-base
@tanstack/preact-ai-devtools
@tanstack/react-ai-devtools
@tanstack/solid-ai-devtools
commit: |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
packages/ai-openai/tests/realtime-token.test.ts (1)
1-56: ⚡ Quick winMove this test next to
src/realtime/token.tsto match test placement rules.The test implementation is good, but this new file is not colocated with the source module it validates. Please place it alongside
packages/ai-openai/src/realtime/token.tsas a*.test.tsfile.As per coding guidelines, "Place unit tests alongside source code in
*.test.tsfiles".🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/ai-openai/tests/realtime-token.test.ts` around lines 1 - 56, Move the test file so it is colocated with the implementation: create packages/ai-openai/src/realtime/token.test.ts containing the current test body, update the imports to import { buildClientSecretRequest, parseClientSecretResponse } from './token' (relative import to token.ts), and remove the original test from packages/ai-openai/tests; ensure the test filename ends with .test.ts and that references to buildClientSecretRequest and parseClientSecretResponse remain unchanged.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/ai-openai/src/realtime/token.ts`:
- Around line 38-55: The parser (parseClientSecretResponse) must fail-fast for
empty token strings and avoid propagating non-string session.model values:
update the validation to require data.value to be a non-empty string (e.g.,
non-whitespace) before returning, and when building config.model only accept
data.session?.model if typeof === 'string' (otherwise use fallbackModel); keep
the existing checks for data.expires_at and preserve provider/token/expiresAt
behavior.
---
Nitpick comments:
In `@packages/ai-openai/tests/realtime-token.test.ts`:
- Around line 1-56: Move the test file so it is colocated with the
implementation: create packages/ai-openai/src/realtime/token.test.ts containing
the current test body, update the imports to import { buildClientSecretRequest,
parseClientSecretResponse } from './token' (relative import to token.ts), and
remove the original test from packages/ai-openai/tests; ensure the test filename
ends with .test.ts and that references to buildClientSecretRequest and
parseClientSecretResponse remain unchanged.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 089bb19e-776a-4c5d-be70-a48a2910c626
📒 Files selected for processing (12)
.changeset/openai-realtime-ga-migration.mddocs/media/realtime-chat.mddocs/reference/functions/realtimeToken.mdexamples/ts-code-mode-web/src/routes/_execute-prompt/api.realtime-token.tsexamples/ts-react-chat/src/lib/use-realtime.tspackages/ai-openai/src/realtime/adapter.tspackages/ai-openai/src/realtime/session-update.tspackages/ai-openai/src/realtime/token.tspackages/ai-openai/src/realtime/types.tspackages/ai-openai/tests/realtime-session-update.test.tspackages/ai-openai/tests/realtime-token.test.tspackages/ai/src/realtime/index.ts
✅ Files skipped from review due to trivial changes (5)
- examples/ts-code-mode-web/src/routes/_execute-prompt/api.realtime-token.ts
- docs/reference/functions/realtimeToken.md
- .changeset/openai-realtime-ga-migration.md
- packages/ai/src/realtime/index.ts
- docs/media/realtime-chat.md
Comment on lines
+38
to
+55
| if ( | ||
| !data || | ||
| typeof data.value !== 'string' || | ||
| typeof data.expires_at !== 'number' || | ||
| !Number.isFinite(data.expires_at) | ||
| ) { | ||
| throw new Error( | ||
| 'OpenAI realtime client secret response missing or malformed `value`/`expires_at`', | ||
| ) | ||
| } | ||
|
|
||
| return { | ||
| provider: 'openai', | ||
| token: data.value, | ||
| expiresAt: data.expires_at * 1000, | ||
| config: { | ||
| model: data.session?.model ?? fallbackModel, | ||
| }, |
Contributor
There was a problem hiding this comment.
Harden response validation for value and session.model.
parseClientSecretResponse currently accepts an empty token string and can propagate a non-string session.model from untrusted API JSON into config.model. This can create late auth/config failures instead of failing fast at parse time.
Proposed fix
if (
!data ||
typeof data.value !== 'string' ||
+ data.value.length === 0 ||
typeof data.expires_at !== 'number' ||
!Number.isFinite(data.expires_at)
) {
throw new Error(
'OpenAI realtime client secret response missing or malformed `value`/`expires_at`',
)
}
+ const resolvedModel =
+ typeof data.session?.model === 'string' && data.session.model.length > 0
+ ? data.session.model
+ : fallbackModel
+
return {
provider: 'openai',
token: data.value,
expiresAt: data.expires_at * 1000,
config: {
- model: data.session?.model ?? fallbackModel,
+ model: resolvedModel,
},
}🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@packages/ai-openai/src/realtime/token.ts` around lines 38 - 55, The parser
(parseClientSecretResponse) must fail-fast for empty token strings and avoid
propagating non-string session.model values: update the validation to require
data.value to be a non-empty string (e.g., non-whitespace) before returning, and
when building config.model only accept data.session?.model if typeof ===
'string' (otherwise use fallbackModel); keep the existing checks for
data.expires_at and preserve provider/token/expiresAt behavior.
The GA realtime API only accepts ['audio'] or ['text'] for output_modalities; the Beta API accepted ['audio', 'text'] and the provider-agnostic RealtimeSessionConfig still legitimately produces it (e.g. the example UI's audio+text mode). Sending both got the whole session.update rejected with: Invalid modalities: ['audio', 'text']. Collapse to ['audio'] when audio is requested — GA audio replies still stream text via response.output_audio_transcript.* events, so visible behavior is unchanged. Live-verified: session.updated accepted. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
tombeckenham
approved these changes
Jun 10, 2026
tombeckenham
left a comment
tombeckenham
left a comment
Contributor
There was a problem hiding this comment.
Thank you. This is approved
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🎯 Changes
Fixes #586
Migrates
@tanstack/ai-openai's realtime adapters from the deprecated OpenAI Beta realtime API (shut down 2026-05-12) to the GA API. The Beta shape caused allopenaiRealtime()connections to fail withbeta_api_shape_disabled.Full GA migration:
openaiRealtime()):POST /v1/realtime/callsinstead of the Beta/v1/realtime?model=...(GA rejects?model=; the model is bound to the ephemeral key).openaiRealtimeToken()):POST /v1/realtime/client_secretsinstead of the retired Beta/v1/realtime/sessions, parsing the GA top-levelvalue/expires_atresponse shape.session.update: GA payload shape via a new purebuildSessionUpdate()helper — requiredsession.type,audio.input.transcription,audio.input.turn_detection,audio.output.voice,output_modalities,max_output_tokens. GA rejects the entire update on any unknown (Beta-named) field, so the original two-line fix alone still left instructions/tools/voice unapplied.temperature(removed in GA) is dropped with a debug log.response.output_audio_transcript.*,response.output_audio.*,output_text/output_audiocontent parts).gpt-realtime; thegpt-4o-(mini-)realtime-previewids (shut down by OpenAI 2026-05-07) are removed fromOpenAIRealtimeModel, docs, and examples.✅ Checklist
pnpm run test:pr.🚀 Release Impact
🧪 Test Plan
realtime-session-update.test.ts(GAsession.updateshape, no Beta field names) andrealtime-token.test.ts(client-secret request/response parsing).client_secrets→ 200 withek_…key;/v1/realtime/calls→ 201 with SDP answer using the ephemeral key;session.updatewith the exact built payload →session.updatedechoing voice, semantic VAD, tools,output_modalities, andmax_output_tokens.🤖 Generated with Claude Code
Summary by CodeRabbit