[TEST — DO NOT MERGE] Demo for #932 secret-coherence pipeline by jottakka · Pull Request #935 · ArcadeAI/docs

jottakka · 2026-04-18T19:06:55Z

⚠️ Testing-only PR — do not merge

Base branch is #932, not main. Close this PR after testing; the phantom secret must not land on main.

What changed compared to the first test run

The first trigger on this branch (#936) produced no cleanup because --skip-unchanged short-circuits before enforceSecretCoherence runs when the Engine API's summary endpoint reports no version change for github. The phantom secret diff was never evaluated.

#932 now has a workflow_dispatch.inputs.providers input that bypasses --skip-unchanged and --all for manual runs. This branch is rebased on that change.

How to run it (updated)

Confirm ANTHROPIC_API_KEY is set as a repo secret. Without it the scanner runs but the LLM edit is skipped.
Go to Actions → Generate toolkit docs → Run workflow.
Pick branch test/secret-coherence-demo.
In the providers input, enter: Github.
Click Run workflow.

The run now uses --providers Github without --skip-unchanged, so the generator processes Github end-to-end — the scanner diffs the committed JSON (with the phantom) against fresh Engine API data (without it), identifies GITHUB_CLASSIC_PERSONAL_ACCESS_TOKEN as removed, scans chunks, and (if the Anthropic key is present) drives Sonnet 4.6 to minimally edit the stale references out.

What you should see in the AUTO PR

In the resulting [AUTO] PR's diff on toolkit-docs-generator/data/toolkits/github.json:

✓ Phantom secret removed from Github.AssignPullRequestUser .secrets / .secretsInfo (that happens automatically because Engine API doesn't return it).
✓ The custom_section chunk (the Secrets Setup section) drops the row / note lines that reference GITHUB_CLASSIC_PERSONAL_ACCESS_TOKEN, leaving the GITHUB_SERVER_URL row intact.
✓ Unrelated chunks (Enterprise Support callout, App Permissions Summary, Configuration & Setup) stay byte-identical — confirms the edit is minimal.
✓ The run log should include a warning like Stale secret reference in toolkit_chunk #N: GITHUB_CLASSIC_PERSONAL_ACCESS_TOKEN.

When done

Close this PR without merging. Delete the branch.

Refs #932

vercel · 2026-04-18T19:07:00Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
docs	Ready	Preview, Comment	Apr 18, 2026 7:55pm

… runs Two workflow additions driven by PR #936 feedback: 1. Job-level `FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"` opts all JavaScript actions into Node 24 ahead of the 2026-06-02 deprecation. actions/checkout@v4, actions/setup-node@v4, peter-evans/create-pull-request@v7, and pnpm/action-setup@v4 all trigger the "Node.js 20 actions are deprecated" annotation today; the opt-in silences it and matches the runtime we'll be forced onto anyway. 2. New `workflow_dispatch` input `providers`. When set to a comma-separated provider list (e.g. "Github"), the run uses `--providers "$providers"` AND drops `--skip-unchanged` so the secret-coherence scan actually re-evaluates those toolkits — even when the Engine API reports no version change. Scheduled and porter_deploy_succeeded runs keep the previous `--all --skip-unchanged` behavior. This is what lets the #935 demo PR actually exercise the pipeline end-to-end: trigger the workflow with `providers=Github` and the phantom secret gets surfaced + cleaned. Tests added: workflow assertions for the new env var and the providers input fallback structure. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

The stale-secret scanner, coverage-gap detector, and summary-generation failures all push warnings onto `result.warnings`. Per-provider mode already echoes those to stdout (line 848 of cli/index.ts). The --all and regenerate-all paths did not — they only appended to the run log file on disk, which GitHub Actions runs don't expose. Result: on the #935 demo, the workflow ran, the phantom secret was removed from the tool's .secrets array, but no cleanup was applied to the stale doc chunk that still referenced it AND there was no signal in the CI log explaining why. The warnings that would have explained "stale secret detected but edit failed" or "stale secret detected but no editor configured" were present in memory but discarded. This commit prints every non-empty `mergeResult.warnings` to stdout right after `mergeAllToolkits()` returns, in both the `generate --all` and `regenerate --all` paths. Format matches existing spinner output: ⚠ Github: 2 warning(s) - Stale secret reference in toolkit_chunk #4: GITHUB_CLASSIC_... - Secret cleanup edit failed for Github (documentation_chunk): ... 551 tests pass, type-check clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…erence Seeds a phantom secret (GITHUB_CLASSIC_PERSONAL_ACCESS_TOKEN) into Github.AssignPullRequestUser's secrets and secretsInfo so that a fresh workflow run against the live Engine API identifies the secret as "removed" when it compares the committed toolkit against the generator output. The existing documentation chunk in github.json still references this same secret name (real-world residue from when the notification tools were dropped in #922), so the secret-coherence step from #932 should: 1. Detect `GITHUB_CLASSIC_PERSONAL_ACCESS_TOKEN` as removed on the `Github.AssignPullRequestUser` tool. 2. Scan chunks and find the matching text in the `custom_section` chunk. 3. Call the Claude Sonnet 4.6 editor to minimally edit that chunk — deleting the table row and note lines that reference the removed secret while preserving the rest of the setup documentation. To test: trigger `Generate toolkit docs` workflow via workflow_dispatch on this branch. Confirm the resulting AUTO PR's diff on github.json: - Removes the phantom secret from the tool's .secrets / .secretsInfo. - Rewrites the Secrets Setup chunk to drop the stale rows. - Leaves unrelated chunk content intact (no re-summarization). Do not merge this commit. The phantom secret must not land on main. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…ts (#932) * feat(toolkit-docs-generator): secret-coherence scan + minimal LLM edits When a toolkit loses a secret upstream (typically because the tool that required it was dropped), the rendered docs can continue to mention that secret in the summary and in hand-authored documentation chunks. One concrete example on main: github.json still references GITHUB_CLASSIC_PERSONAL_ACCESS_TOKEN after the notification tools were removed in PR #922. Symmetrically, toolkits can end up with current secrets that the summary never mentions, or mention secrets without any link to the Arcade config docs — leaving readers without the information needed to actually configure them. This adds a two-stage pipeline that runs after summary generation: 1. Deterministic scanners (src/merger/secret-coherence.ts) - detectStaleSecretReferences: diffs current vs previous toolkit secret sets and scans summary, toolkit chunks, and per-tool chunks by exact substring for each removed secret. - detectSecretCoverageGaps: flags current secrets missing from the summary and a missing link to the Arcade secret config docs. - groupStaleRefsByTarget: collapses multiple removed-secret hits in the same artifact into a single edit target so the LLM is called at most once per (summary | chunk). 2. Targeted LLM editor (src/llm/secret-edit-generator.ts) - Unlike toolkit-summary-generator (which rewrites from scratch and tends to oversimplify), this editor is prompted to make the smallest possible change: delete sentences/rows that are only about the removed secret, minimally rewrite any sentence that mentions the removed secret alongside other content, and never re-summarize or reorder sections. - A separate fillCoverageGaps method adds missing secret mentions and, when required, the Arcade config docs link — also without rewriting unrelated text. Both steps are wired into DataMerger.enforceSecretCoherence, called after maybeGenerateSummary. The editor is optional: if unconfigured, the scanners still run and emit warnings, but no content is rewritten. Failures in the editor are caught and surfaced as warnings so a single LLM error does not break the run. Wiring changes: - DataMergerConfig gains an optional secretEditGenerator. - CLI gains --llm-editor-provider / --llm-editor-model / --llm-editor-api-key / --llm-editor-base-url / etc., mirrored by LLM_EDITOR_* env vars, with --skip-secret-coherence for the scan-and-edit step. Resolver fails open: a missing API key degrades to scanner-only warnings instead of crashing the run. - Workflow generate-toolkit-docs.yml now passes editor flags pointing at Anthropic + claude-sonnet-4-6 (overridable via secrets) so the editor stays on a stronger model than the gpt-4o-mini used for bulk summary and example generation. Summary prompt updates (src/llm/toolkit-summary-generator.ts): - Drop the hard 60–140 word cap; ask for "compact but complete". - Require each current secret be named in backticks with a one-line factual description of how to obtain it from the provider. - Require the Arcade secret config docs link at the end of the **Secrets** section. Tests: - tests/merger/secret-coherence.test.ts (13 tests) covers scanner behavior across summary, toolkit chunks, tool chunks, coverage gaps, and target grouping. - tests/llm/secret-edit-generator.test.ts (6 tests) exercises the cleanup/coverage flows and the fence-stripping / empty-response guards with a mocked LLM client. - Two new DataMerger integration tests verify that a removed secret surfacing in a doc chunk drives exactly one cleanup call and that the editor-disabled path still emits the warning. - tests/workflows/generate-toolkit-docs.test.ts asserts the new editor flags are present in CI. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(toolkit-docs-generator): address ACR findings on secret coherence Two issues surfaced by `/acr-run`: 1. FENCE_PATTERN (secret-edit-generator.ts) was non-greedy and unanchored, so stripOptionalFence stopped at the FIRST inner ``` when the LLM wrapped its edit in a markdown fence and the edit itself contained a fenced code block. Result: the rest of the edit was silently dropped with no error — corrupted doc chunks written to disk. Fix: anchor the pattern to ^…$ and use a greedy capture so the match extends to the outer closing fence. 2. enforceSecretCoherence (data-merger.ts) computed coverage gaps once, before stale cleanup ran. If cleanup modifies the summary and incidentally drops a passage that mentioned a current secret, the pre-cleanup gap snapshot would miss it. Fix: re-run detectSecretCoherenceIssues after applyStaleRefCleanup so the coverage fill sees post-cleanup state. Tests: - Two new fence tests cover (a) preserving inner code blocks when unwrapping the outer fence, and (b) leaving unwrapped responses with inner blocks untouched. - One new DataMerger test proves the coverage editor receives post- cleanup summary content (not a stale snapshot). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore(toolkit-docs-generator): raise editor max-tokens default to 8192 4096 was tight. Largest single artifact in current data is a ~6K-char doc chunk (googlenews) ≈ 1.5K output tokens for a minimal-edit rewrite; a summary with no word cap for a 40+ tool toolkit with several secrets can land in the 2–3K output-token range. 8K gives comfortable margin without meaningful cost or latency impact on Sonnet 4.6. Help text updated to match. Callers can still override via --llm-editor-max-tokens. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(toolkit-docs-generator): document secret coherence + loosen per-secret prose cap Two changes: 1. README: new Secret coherence section covering the scan/edit pipeline, the editor CLI flags, the claude-sonnet-4-6 default, fail-open behavior when no API key is set, and a local invocation example. Required/optional CI secrets updated with ANTHROPIC_API_KEY and ANTHROPIC_EDITOR_MODEL. Key CLI options list updated with the new flags. 2. Prompts (summary generator + coverage-fill editor) no longer cap each secret at one line. Instead they ask for as much detail as the secret actually needs — a short URL override may be one line; a scoped API key typically needs several sentences naming the provider dashboard page, required scopes or permissions, and any account tier. Both prompts also request an inline markdown link to the provider's own docs page for how to create/retrieve the secret when the model knows it, and explicitly forbid inventing URLs. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(toolkit-docs-generator): summary prompt no longer repeats OAuth scopes Per follow-up on PRs #928 and #929, the OAuth section of each summary should name the provider and link to the Arcade per-provider auth docs rather than enumerate scopes. Scopes already live on the provider reference page and repeating them in toolkit summaries creates drift every time a provider page updates. Changes: - Add ARCADE_AUTH_PROVIDERS_BASE_URL constant alongside the existing Arcade secret URLs in secret-coherence.ts. - Rewrite the OAuth bullet in toolkit-summary-generator.ts's prompt to require a link to {base}/<providerId> and explicitly forbid listing scopes. - Drop scopes from formatAuth's prompt payload so the model has no stray scope list to fall back on. - README: note the no-scopes-in-summary rule and point to the provider reference pages as the source of truth. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(toolkit-docs-generator): address ACR findings on secret coherence (round 2) Four findings from /acr-run: 1. HIGH (5/5) — ANTHROPIC_EDITOR_MODEL was documented as a fallback env var in the README but never read by resolveSecretEditGenerator. A local dev setting only ANTHROPIC_EDITOR_MODEL would get `model = undefined`, the (provider && model) guard would fire, and the editor would silently stay inactive. Extract resolveEditorModel helper that walks `--llm-editor-model` → LLM_EDITOR_MODEL → ANTHROPIC_EDITOR_MODEL in documented order, and use it from both the resolver and the verbose-log blocks. 2. MEDIUM — --skip-secret-coherence was documented to "disable both the scan and the edit step entirely" but DataMerger never received the flag; enforceSecretCoherence always ran, so coherence warnings still appeared when the user explicitly opted out. Add `skipSecretCoherence` to DataMergerConfig, gate enforcement on it, and pass it through from all three merger construction sites in the CLI. 3. MEDIUM — FENCE_PATTERN matched non-markdown language fences (```python, ```bash, ```json). A documentation chunk whose content was a code block would have its fences stripped, corrupting the edited output. Tightened the pattern to require either an empty, markdown, md, or text tag followed by a newline between the opening fence and the captured content, so language-tagged code blocks fall through stripOptionalFence unchanged. 4. LOW — verbose log showed "model: undefined" when only ANTHROPIC_EDITOR_MODEL was set. Fixed by #1. Tests added: - fence strip preserves `\`\`\`python` and `\`\`\`bash` code blocks verbatim - skipSecretCoherence suppresses both edits and warnings 549 tests pass, type-check clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(workflow): Node 24 opt-in + focused workflow_dispatch for manual runs Two workflow additions driven by PR #936 feedback: 1. Job-level `FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"` opts all JavaScript actions into Node 24 ahead of the 2026-06-02 deprecation. actions/checkout@v4, actions/setup-node@v4, peter-evans/create-pull-request@v7, and pnpm/action-setup@v4 all trigger the "Node.js 20 actions are deprecated" annotation today; the opt-in silences it and matches the runtime we'll be forced onto anyway. 2. New `workflow_dispatch` input `providers`. When set to a comma-separated provider list (e.g. "Github"), the run uses `--providers "$providers"` AND drops `--skip-unchanged` so the secret-coherence scan actually re-evaluates those toolkits — even when the Engine API reports no version change. Scheduled and porter_deploy_succeeded runs keep the previous `--all --skip-unchanged` behavior. This is what lets the #935 demo PR actually exercise the pipeline end-to-end: trigger the workflow with `providers=Github` and the phantom secret gets surfaced + cleaned. Tests added: workflow assertions for the new env var and the providers input fallback structure. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(cli): surface per-toolkit merge warnings to stdout in --all runs The stale-secret scanner, coverage-gap detector, and summary-generation failures all push warnings onto `result.warnings`. Per-provider mode already echoes those to stdout (line 848 of cli/index.ts). The --all and regenerate-all paths did not — they only appended to the run log file on disk, which GitHub Actions runs don't expose. Result: on the #935 demo, the workflow ran, the phantom secret was removed from the tool's .secrets array, but no cleanup was applied to the stale doc chunk that still referenced it AND there was no signal in the CI log explaining why. The warnings that would have explained "stale secret detected but edit failed" or "stale secret detected but no editor configured" were present in memory but discarded. This commit prints every non-empty `mergeResult.warnings` to stdout right after `mergeAllToolkits()` returns, in both the `generate --all` and `regenerate --all` paths. Format matches existing spinner output: ⚠ Github: 2 warning(s) - Stale secret reference in toolkit_chunk #4: GITHUB_CLASSIC_... - Secret cleanup edit failed for Github (documentation_chunk): ... 551 tests pass, type-check clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(workflow): remove focused providers dispatch input Simplify manual toolkit docs runs by removing the workflow_dispatch providers override and restoring the default full run path with --all --skip-unchanged. Made-with: Cursor * fix(toolkit-docs-generator): address cursor review on secret coherence Reuse shared secret collection logic across merger modules and restore stale-summary coverage in data-merger tests. Made-with: Cursor * fix(cli): use editor flag in provider validation message Pass the editor-specific option name to provider validation so invalid --llm-editor-provider values return actionable guidance. Made-with: Cursor --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

jottakka self-assigned this Apr 18, 2026

vercel Bot deployed to Preview April 18, 2026 19:10 View deployment

jottakka force-pushed the test/secret-coherence-demo branch from 5638ef3 to ac00598 Compare April 18, 2026 19:17

vercel Bot deployed to Preview April 18, 2026 19:18 View deployment

jottakka force-pushed the test/secret-coherence-demo branch from ad7b8c8 to c58424f Compare April 18, 2026 19:53

vercel Bot deployed to Preview April 18, 2026 19:55 View deployment

jottakka closed this Apr 18, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[TEST — DO NOT MERGE] Demo for #932 secret-coherence pipeline#935

[TEST — DO NOT MERGE] Demo for #932 secret-coherence pipeline#935
jottakka wants to merge 1 commit intofeat/toolkit-docs-secret-coherencefrom
test/secret-coherence-demo

jottakka commented Apr 18, 2026 •

edited

Loading

Uh oh!

vercel Bot commented Apr 18, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

jottakka commented Apr 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ Testing-only PR — do not merge

What changed compared to the first test run

How to run it (updated)

What you should see in the AUTO PR

When done

Uh oh!

vercel Bot commented Apr 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

jottakka commented Apr 18, 2026 •

edited

Loading

vercel Bot commented Apr 18, 2026 •

edited

Loading