feat: AI Generate — OpenAI-backed agent loop for widget creation#11
Open
feat: AI Generate — OpenAI-backed agent loop for widget creation#11
Conversation
Plan for a settings-configured OpenAI key plus an AI-driven widget creation flow that runs an agent loop (generate → run in JSX runtime → feed errors back → regenerate) before handing the result to the user for review, preview, and save. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Shared: AISettings store, SwiftOpenAI-backed AIClient, reference snapshot builder (samples Script.bundle for prompt context), PromptBuilder, AgentRuntimeBridge, AgentLoop, AIGenerateSession, cross-platform AIGenerateProgressView. - iOS: SettingAIView, AIGenerateView and AIReviewView; wired into SettingsView (new AI group) and CreateGuideView (new "Generate with AI" entry). - macOS: SettingAIView (hosted in Settings scene, Cmd+,), AIGenerateWindowView (split-view sheet with prompt, live progress, preview, refine, save), and a sparkles toolbar button in SidebarView. Project file (.pbxproj) registration and SwiftOpenAI SPM dependency follow in the next commit. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Wires the new AI layer into both Xcode projects: - Adds the SwiftOpenAI Swift Package (from: 4.4.9) to the ScriptWidget and ScriptWidgetMac main-app targets. Widget and share extensions intentionally do NOT link it. - Creates a new "AI" group under Shared/ScriptWidgetRuntime pointing at ../Shared/ScriptWidgetRuntime/AI and registers the 8 shared Swift files in both projects. - iOS: registers SettingAIView, plus a new "AIGenerate" group with AIGenerateView and AIReviewView. - macOS: registers new "Settings" and "AIGenerate" groups holding SettingAIView and AIGenerateWindowView respectively. - Adjusts AIClient to the current SwiftOpenAI API: the custom-baseURL path uses overrideBaseURL (String-based apiKey), and response `choices`/`message` are now optionals. iOS: ScriptWidget, ScriptWidgetWidget, and ScriptWidgetShare all build cleanly. macOS: pre-existing Vapor/MultipartKit/swift-nio incompatibility with the current Xcode SDK blocks building ScriptWidgetMac for reasons orthogonal to this feature (the unpatched project.pbxproj fails with the same errors on a clean build). Resolving that is outside the scope of this PR. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
everettjf
changed the title
Adds AIExamplePrompts with 8 curated starters (Weather, Clock, Countdown, Crypto Price, Battery Ring, Quote, Steps, Habit Grid) that each describe concrete colors, data sources, and layout so the agent loop converges quickly. Surfaced as a horizontal chip row on both iOS AIGenerateView and macOS AIGenerateWindowView; tapping a chip fills the prompt field and sets the recommended widget size. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…menu command Closes remaining parity gaps with the iOS flow: - Shows a "Did not fully converge" banner above the preview when the agent loop exhausts its iteration budget, matching AIReviewView on iOS. - Prefills the save-name field with "AI yyyy-MM-dd HHmm" on open and whenever the generated JSX changes (only if still empty), so the Save button is reachable without typing. - Adds File → "Generate Widget with AI..." with the ⌘⇧N shortcut, routed through a notification so SidebarView handles the not-configured case consistently with the toolbar button. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Vapor's only job on macOS was to spin up a localhost HTTP server on port 23355 that served Monaco editor static files from Editor.bundle/static. That came with a large transitive dep graph (swift-nio, MultipartKit, WebSocketKit, etc.), and the version combination compatible with Vapor's open `from: 4.0.0` range no longer builds against the current Xcode SDK under Swift 6 strict concurrency — MultipartKit's retroactive Sendable conformance on FormData types is rejected by the compiler, and older swift-nio pins fail CNIODarwin module imports. This replaces the whole HTTP server with a tiny WKURLSchemeHandler: - Adds EditorSchemeHandler that serves the `scriptwidget-editor://` scheme directly from Editor.bundle/static (with a path-traversal guard and MIME-type mapping). - Registers the handler on WKWebViewConfiguration in EditorInternalWebView.init(). - editorWebServiceUrl() now returns a scriptwidget-editor:// URL. - AppDelegate no longer starts a background HTTP server. - Removes Vapor from the project's SPM graph (XCRemoteSwiftPackage Reference, product dependency, packageProductDependencies entry, and Frameworks build-file entry). Result: ScriptWidgetMac and ScriptWidgetMacWidget both BUILD SUCCEEDED under the current Xcode, and 20+ transitive packages (Vapor, MultipartKit, WebSocketKit, RoutingKit, ConsoleKit, AsyncKit, swift-crypto, swift-certificates, swift-asn1) drop out of the dep graph entirely. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The sparkles toolbar button was easy to miss. Bring the AI entry
point into the same flow iOS uses — first thing users see when they
open the create sheet.
- macOS CreateGuideView now opens with a prominent "Generate with
AI" card on top of the existing blank-widget name input. Tapping
it dismisses the sheet and posts the open-request notification so
SidebarView handles the not-configured alert path uniformly.
- Promotes the sidebar toolbar button: now uses `wand.and.stars` with
a Label ("Generate with AI" + icon) so it shows a text affordance
instead of a bare symbol, placed before the "+" button. Adds ⌘⇧N
hint in the help tooltip.
- Relabels the existing blank-create button to "Create Blank" so the
choice between blank vs AI is obvious.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Lower the ramp-up cost for new users by packaging the 45 bundled templates as a browsable gallery with categories, a first-run guide, and a one-tap way to fork existing widgets. - Add ScriptMetadata (category / tags / difficulty / icon / featured) loaded from per-template meta.json; 44 meta files added. - Replace flat "Create from template" list with a searchable grid + category chips on both iOS and macOS. macOS gains a full gallery (was AI + Blank only). - Replace empty-state placeholders with onboarding: hero, how-it-works, featured templates. - Add "Remix" (duplicate) — iOS swipe action, macOS context menu — backed by ScriptManager.duplicateScript. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replace the single AI-config blob with a list of named profiles (host, model, key, auth method) plus an active-profile pointer, all stored in the app-group UserDefaults under ai.profiles.v2 with a one-shot migration from the legacy single-profile keys. Add an OpenAI OAuth (PKCE) sign-in path as an alternative to API keys, ported from openrocky: Codex CLI client_id, local 127.0.0.1 HTTP listener for the redirect, JWT-decoded chatgpt_account_id, keychain-backed credential vault with refresh-on-expiry. Settings UI rebuilt to manage profiles (sidebar/detail on macOS, list+push-editor on iOS) with provider preset chips, model suggestions, segmented API Key | OAuth picker, per-profile test connection. Generate sheet gains a profile picker when more than one profile exists. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The apiKey field on AIProfile was being JSON-encoded into the app-group UserDefaults alongside non-secret fields. Move it into the Keychain (account scriptwidget.profile.apiKey.<id>) and hydrate it back onto AIProfile on read. Encoding now omits apiKey; the decoder still accepts it as a legacy fallback during migration. Migration runs once per install, gated by ai.profiles.apiKeyKeychainMigrated.v1: existing JSON profiles are decoded with apiKey intact, then re-saved through the new path which writes the keys to Keychain and rewrites the JSON without them. Update settings copy on both targets to match — drop the orange "plain-text UserDefaults" warning. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Fix the deprecation warnings that have a one-line, behavior-preserving
replacement:
- onChange(of:perform:) → zero/two-arg closure form (deployment
targets are iOS 18 and macOS 14, both ≥17/14).
- String(contentsOfFile:) → with explicit utf8 encoding.
- NSLocale.current.languageCode → Locale.current.language.languageCode.identifier.
- kUTTypeURL / CoreServices → UTType.url / UniformTypeIdentifiers.
- Text + Text(Image(...)) + Text → string interpolation.
- Move CLLocationManager guards under #else so the IsWidgetTarget
branch's `return` no longer leaves dead code on the other path.
Skipped (need their own commits or architectural decisions):
- UIScreen.main inside the bundled TOCropViewController fork.
- NavigationLink(isActive:destination:label:) → NavigationStack
migration.
- Sendable / strict concurrency warnings on ScriptManager and AnyShape.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Per-execution state (package handle + console logger) used to live in a process-wide `var sharedRunningState`. Two paths could touch it concurrently — the editor preview and the AI agent loop — and would overwrite each other's logger and package, mixing log lines or stealing the file root mid-run. AgentRuntimeBridge worked around this with a serial DispatchQueue, but the editor preview path didn't, so a preview running while AI iterated could still race. Hang the running state off the owning JSContext via an associated object. JSExport callbacks (`$file`, `$console`) read it back through JSContext.current(), so each execution sees only its own state. Drop the global, drop the AI bridge's serial queue (now genuinely safe to run concurrent generations), and switch the iOS / macOS preview data objects to read logs from `runtime.runningState?.logger.logs`. The preview paths now also keep their runtime alive on failure so logs remain readable after a failed run (matching prior behavior). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The AI feature has not shipped yet (still on feature/ai-generate),
so there is no installed user base whose apiKey lives in
UserDefaults. Drop both migration paths:
- migrateLegacyIfNeeded (legacy ai.apiKey/ai.baseURL/ai.model →
Default profile)
- migrateAPIKeysToKeychainIfNeeded (existing JSON profiles
carrying apiKey → Keychain)
Also remove the related UserDefaults keys (legacyAPIKey, legacyBaseURL,
legacyModel, apiKeyKeychainMigratedFlag) and the legacy `case apiKey`
in AIProfile's CodingKeys (apiKey was decoded as a fallback). The
decoder now sets apiKey to "" and AISettingsStore hydrates it from
Keychain on load.
First-launch behavior is unchanged: loadProfiles() seeds one default
profile when ai.profiles.v2 is absent.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds the AI Generate feature end-to-end: users configure an OpenAI (or compatible) API key in settings, describe a widget in natural language, and an agent loop iterates generate → run in the JSX runtime → feed errors back → regenerate until the widget renders cleanly. User then reviews a live preview, optionally refines it with another prompt, and saves into the Scripts library.
Design doc is in
docs/ai-generate.md(first commit on this branch).What's included
Shared layer (
Shared/ScriptWidgetRuntime/AI/)AISettings.swiftgroup.everettjf.scriptwidgetapp group.AIReferenceSnapshot.swiftScript.bundle/component/*andapi/*at startup so the system prompt always teaches the current DSL surface.PromptBuilder.swiftAIWidgetSizeenum + system prompt rules + first-turn / fix-turn / refine-turn user prompt templates + fenced-code stripper.AIClient.swiftoverrideBaseURLfor custom endpoints so Azure / DeepSeek / Ollama work.AgentRuntimeBridge.swiftScriptWidgetPackageand drivesexecuteJSXSyncForWidget, capturing console logs + typed errors.AgentLoop.swift[system, firstUser, lastAssistant, lastUserFix]per turn so context stays bounded.AIGenerateSession.swift@MainActorObservableObjectstate machine:.idle / .thinking / .running / .fixing / .done / .exhausted / .failed / .cancelled.AIGenerateProgressView.swiftiOS —
SettingAIView(under Settings → new "AI" group), newAIGenerate/directory withAIGenerateView+AIReviewView, and a "✨ Generate with AI" entry at the top ofCreateGuideView.macOS — new
Settings/SettingAIViewhosted in the standardSettings {}scene (⌘,), newAIGenerate/AIGenerateWindowViewwith split-view layout (prompt + progress + preview + refine), and asparklestoolbar button inSidebarView.Xcode projects — SwiftOpenAI SPM dep added to both projects' main app targets only (widget / share extensions are not linked). New files registered in Sources build phases; file references grouped under a new logical
AIgroup pointing at../Shared/ScriptWidgetRuntime/AI.Behavior
gpt-4o-mini, max 20 iterations (user-adjustable 5–100),https://api.openai.com, temp 0.7.baseURL; trailing/v1or/is normalized off.nil, element is not a#UI Not Found#/#Failed#/#Loading#fallback, and logs contain no[error]/uncaughtmarkers.sharedScriptManager.createScript(...)to commit into the real Scripts library and postsscriptCreateNotificationso the home list refreshes.Build status
ScriptWidget,ScriptWidgetWidget,ScriptWidgetShareall build cleanly oniphonesimulatorwith the current Xcode.ScriptWidgetMachits a pre-existing dependency-graph failure (swift-nio/MultipartKitunsendable-userInfo errors) that also reproduces on the unpatchedmaintree under the same Xcode version. Root cause is Vapor's open version range pulling packages that no longer compile under current Swift strict-concurrency. That is orthogonal to this feature and should be addressed in a separate PR (e.g. pin Vapor to a newer release or replace the Monaco editor web service with a non-Vapor static file server).Test plan
open iOS/ScriptWidget.xcodeproj→ pickScriptWidgetscheme → build + run.🤖 Generated with Claude Code