Add daily upstream-sweep routine and three rounds of hardening#29
Merged
JimCollinson merged 6 commits intomainfrom May 11, 2026
Merged
Add daily upstream-sweep routine and three rounds of hardening#29JimCollinson merged 6 commits intomainfrom
JimCollinson merged 6 commits intomainfrom
Conversation
…hecks Adds the daily upstream-drift sweep routine described in planning/routines/upstream-sweep.md. The routine is hosted on Claude Desktop (Remote variant), runs once per day, calls a deterministic Python scanner to detect drift against pinned upstream SHAs, audits each affected page against the exact head_sha the scanner found, and opens metadata-only sweep PRs or prose draft PRs per the documented topology. Scanner (scripts/sweep_poll.py): - walks every <!-- verification: --> block in docs/**/*.md, every entry in skills/start/version.json's verified_commits, and every entry in SKILL.md frontmatter's verified_commits - resolves (repo, ref) via repo-registry.yml + GitHub default_branch - emits a per-record JSON drift report on stdout - fail-closed on auth / network / parse / shape / unknown-mode errors - target-manifest blocks deliberately skipped (separate JSON array) Required checks for branch protection: - sweep-guard: validates the metadata-only envelope on claude/sweep-* branches; verifies docs diffs lie inside verification blocks and only touch source_commit / verified_date lines, version.json only changes verified_commits, SKILL.md frontmatter only changes verified_commits and verified_date, CHANGELOG is untouched, and exactly one planning/sweeps/<branch-date>.md is added. - sweep-sha-reachability: validates every recorded SHA against the upstream ref via the GitHub commits + compare APIs; runs on both claude/sweep-* and claude/prose-* branches; widens the scope to every changed verification block (catches date-only refreshes whose existing SHA was previously broken). Both workflows use a real gate step (id: scope) plus an `if:` on every later step — exit-0 from an early step does not skip later steps in GitHub Actions. skills/start/MAINTAINING.md: tighten the patch-bump rule. Releases (version bumps) move SKILL.md + version.json + CHANGELOG together; stamp refreshes are a separate gesture that touches only verified_commits in both files plus verified_date in SKILL.md. Auto-merge is deferred to v1.5; v1 ships scheduled detection, per-page audit, and PR/issue creation only.
…ked key sets, POSIX bootstrap - Add prose-guard.yml enforcing the claude/prose-* envelope and the SKILL.md linked-release rule (skill body changes ⇒ all five release fields must move; body unchanged ⇒ none of them may). - Pin PyYAML literally in sweep-guard.yml, sweep-sha-reachability.yml, and prose-guard.yml; the requirements file is PR-controlled and cannot be the install source. - Lock verified_commits key sets in version.json and SKILL.md frontmatter on both sweep-guard and prose-guard so the routine can refresh values but never silently add or remove repos. - Forbid scripts/, .github/, repo-registry.yml, and component-registry.yml on both guards so a hand-rolled PR cannot widen the routine's surface. - Rewrite Step 0 of the routine prompt in POSIX shell with capture-then-decide (no mapfile, no Bash arrays, no gh issue create --json/--jq). - Expand Step 4 of the prompt and the policy doc into the explicit nine-step Opus 4.7+ audit/write/verify loop, the page batching rule, the audit-diff fetch rule (fetch both recorded_sha and head_sha; fall back to the compare API; never compare against a moving branch), and a concise PR body format. - Note the linked-release coupling in MAINTAINING.md so a manual reviewer knows the prose PR carries the full skill release set.
- prose-guard: positive path allowlist; raw-value verified_commits type
check (no `... or {}` coercion); target-manifest protection on docs
blocks (multiset of literal block bodies) and on version.json/SKILL.md
(structural equality on verification_mode + verified_commits when
either side is target-manifest); explicit `status` immutability;
`-draft` suffix preservation; published_date/verified_date must agree
on linked release; sweep summary must be added (A), not modified.
- sweep-guard: same target-manifest protection on docs/version.json/
SKILL.md; raw-value verified_commits type check; relaxed branch regex
to allow optional `-<slug>` suffix matching the prose form.
- sweep-sha-reachability: emit informational summary lines for skipped
target-manifest entries instead of silently dropping them.
- upstream-sweep.md / upstream-sweep-prompt.md: --limit 1000 on the
open-PR collision check so a busy repo cannot hide a still-open prior
PR behind gh's default 30-row cap; five-case page batching rule with
the deferred-record self-check covering the entire verification block;
manual-review issue de-duplication via fingerprint client-side
matching (no `gh issue --search`); Step 5 ordering (issues first, PRs
second, backlinks third) so PR bodies can reference issue numbers and
reused issues accumulate a chronological run trail.
Five defects found during synthetic verification:
1. sweep-sha-reachability.yml crashed with AttributeError when
verified_commits was a list (no .items()). Added explicit raw type
checks before iteration in both call sites so the workflow fails
closed with a clean message instead of a Python traceback.
2. sweep_poll.py used `data.get("verified_commits") or {}` which
silently coerced missing/null/empty-list values to an empty map,
skipping skill verification records. Replaced with explicit
presence + isinstance(dict) checks at both call sites.
3. upstream-sweep-prompt.md Step 0 only created the
upstream-sweep-status label. Step 5 then opened manual-review
issues with --label upstream-sweep-manual-review and would 404 on
first run. Bootstrap now creates both labels. Added --limit 1000
to the gh label list call too.
4. prose-guard.yml only recorded the new path for rename/copy diffs,
so a claude/prose-* PR could rename a forbidden old path into an
allowed new one and slip past the envelope check. Now records
both old and new paths, mirroring sweep-guard.
5. Both guards protected target-manifest verification_mode and
verified_commits but not verified_date. Added verified_date to
all three target-manifest immutability checks (version.json in
prose-guard, SKILL.md in prose-guard, SKILL.md and version.json
in sweep-guard).
Three review findings on top of ddeb45a: 1. validate_sha() in sweep-sha-reachability.yml could still hit sha[:7] when an entry of verified_commits had a non-string value (null, int, list). Added an isinstance(sha, str) guard at the top of validate_sha so all three call sites (docs blocks, version.json, SKILL.md) fail closed with a clean message instead of a traceback. Mirrored with per-entry checks in sweep_poll.py walk_version_json and walk_skill_md. 2. parse_skill_md_frontmatter used `yaml.safe_load(raw) or {}` which coerced a present-but-empty (or null) frontmatter into {}, then walk_skill_md returned early on `if not fm`, silently skipping SKILL.md verification. Now: parse returns None only for an absent file and raises FailClosed for any present-but-shape-invalid case. walk_skill_md uses `fm is None` instead of falsiness, so an explicitly empty {} frontmatter falls through to the existing verification_mode / verified_commits checks and fails closed. 3. The audit fetch step in upstream-sweep-prompt.md used cd "$TMP" without a return, so a tired or literal agent could continue later PR-writing steps from inside the upstream checkout. Changed to git -C "$TMP" so cwd never moves, with explicit guidance to wrap any genuinely-cwd-needing follow-up in a subshell.
Review pass 5: the F3 fix in 8c541eb fixed Step 4.1 but left Step 4.2 and the policy doc's fetch block running bare git commands. A literal routine run would have inspected the docs repo (the cwd of the routine) instead of the upstream checkout, silently producing misleading audit input. Now every git command in the audit step carries -C "$TMP": - planning/routines/upstream-sweep-prompt.md §4.2: log/diff/diff - planning/routines/upstream-sweep.md §Audit-diff fetch rule: full fetch block + the inline mention of log/diff - planning/routines/upstream-sweep.md §Audit/write/verify summary L198: mirrors the prompt's command form Added an explicit anti-pattern note in the prompt: a bare git log or git diff here would inspect the docs repo, not the upstream tree.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds the daily upstream-drift sweep routine — scanner, Claude Desktop prompt, sweep-guard / prose-guard / sweep-sha-reachability required checks — and three rounds of hardening discovered while building synthetic verification PRs against the routine.
The full path of the branch is:
a6bce6b— initial routine (scanner, prompt, policy, three required checks)e5869c5— v1.1 hardening (prose-guard, pinned deps, locked key sets, POSIX bootstrap)7b23c92— core hardening fixes (positive envelope allowlist, target-manifest mode-flip protection, status immutability, raw-value type checks, branch regex relaxation,--limit 1000, manual-review de-dup ordering, deferred-record self-check broadening)ddeb45a— final hardening this round (five defects below)Net effect: the routine is fail-closed on every metadata shape the scanner can encounter, both guards reject silent target-manifest flips on either
version.jsonorskills/start/SKILL.md, and the routine bootstrap can no longer 404 on its own labels.What
ddeb45afixes (most recent commit; the rest is in earlier commits).github/workflows/sweep-sha-reachability.ymlobj.get("verified_commits") or {}followed by.items()raisedAttributeError: 'list' object has no attribute 'items'when a sweep PR landedverified_commits: [...]. Replaced with explicitisinstance(commits, dict)checks at both call sites (version.json, SKILL.md).scripts/sweep_poll.pyor {}coercion silently swallowed missing-key,null, and empty-list cases at both call sites. Replaced with explicit presence +isinstance(dict)checks wrapped inFailClosed.planning/routines/upstream-sweep-prompt.md,planning/routines/upstream-sweep.mdupstream-sweep-status. Step 5 then opened manual-review issues with--label upstream-sweep-manual-reviewand would 404 on the routine's first ever run. Bootstrap now creates both labels;gh label listuses--limit 1000..github/workflows/prose-guard.ymlclaude/prose-*PR could rename a forbidden old path into an allowed new prefix and slip past the envelope allowlist. Now records both old and new paths, mirroringsweep-guard.yml..github/workflows/prose-guard.yml,.github/workflows/sweep-guard.ymlverification_modeandverified_commitsbut notverified_date. Addedverified_dateto all three TM-protected predicates (version.json in prose-guard, SKILL.md in prose-guard, SKILL.md and version.json in sweep-guard).Synthetic verification
Five tests against
claude/update-autonomy-docs-RNl36before this round'sddeb45ahardening, plus one rerun after:A)T5 also surfaced the
sweep-sha-reachabilityAttributeError now fixed by P1. Rerun against the post-ddeb45aHEAD (PR #28, since closed):sweep-guard: failure (clean)sweep-sha-reachability: failure with the new readable message —skills/start/version.json:verified_commits: missing or not a map/object— no Python tracebackprose-guard: success (correctly green-skips onclaude/sweep-*head ref)T1–T4 were not rerun. The patches in
ddeb45aonly add new fail-closed paths; they don't loosen any rule those tests exercise.Test plan
head_refdoes not matchclaude/sweep-*orclaude/prose-*, so all three required checks should pass via their gate steps.isinstancechecks.Out of scope
.gitignorecovering__pycache__/is good hygiene and will be filed as a small follow-up rather than bundled here.