[docs-noob-tester] 📚 Documentation Noob Test Report - 2026-04-23

[github-actions[bot]]

Summary

• Date of test: 2026-04-23
• Pages visited: Home (/gh-aw/), Quick Start (/gh-aw/setup/quick-start/), CLI Commands (/gh-aw/setup/cli/)
• Overall impression: The homepage immediately communicates what the tool does and the security story is compelling. The Quick Start is well-structured with clear steps, but a first-timer will hit several stumbling blocks around AI account setup, the githubnext/agentics reference, and the concept of "frontmatter" introduced without explanation.

⚠️ Note: Playwright screenshots were unavailable due to network isolation between the agent container and Playwright container. Content was analyzed via curl.

---

🔴 Critical Issues Found

1. githubnext/agentics repository is not publicly documented

In Quick Start Step 2, users run:

gh aw add-wizard githubnext/agentics/daily-repo-status

There is no mention of whether githubnext/agentics is public, how to browse available workflows, or what daily-repo-status actually is before running the command. A new user has no way to verify this will work or discover other available workflow names. If this repo is private or renamed, the guide silently fails with no recovery path.

Fix: Link directly to the githubnext/agentics repo, list a few available workflow names, and add a note if the repo has access restrictions.

2. "COPILOT_GITHUB_TOKEN" secret setup is opaque

Step 2 mentions setting up a COPILOT_GITHUB_TOKEN which is described as "a separate GitHub token with Copilot access — distinct from the default GITHUB_TOKEN". This is a major blocker for new users. The only guidance is a link to an "Authentication" page — there's no inline summary of what scopes are needed or how to create this token.

Fix: Add a collapsible inline box in Step 2 with a 3-step summary of how to create the token (Settings → Developer Settings → PAT → scopes).

3. "Frontmatter" jargon introduced without definition

Step 4 says "If you have changed the frontmatter (the YAML configuration block between --- markers...)". This is the first mention of frontmatter in the Quick Start. A beginner programmer may know what YAML is, but "frontmatter" is a static-site-generator term that GitHub Actions users won't know. The parenthetical definition is buried mid-sentence.

Fix: Add a callout box at the top of Step 4 or a tooltip: "Frontmatter is the YAML configuration block at the top of your .md file, wrapped in --- markers. It controls how the workflow runs."

---

🟡 Confusing Areas

4. "Peli's Agent Factory" in the navbar — who is Peli?

The top navigation shows: Quick Start | Create | Examples | Docs | FAQ | Blog | **Peli's Agent Factory**. As a new user I have no idea who "Peli" is or why this link is in the primary navigation of an official tool. It looks like a personal page accidentally left in the nav, which damages trust in the documentation.

Fix: Either remove this from the main nav or rename it to something that makes sense without prior context (e.g., "Workflow Gallery").

5. Homepage video is a fallback

The Quick Start page contains: Your browser doesn't support HTML5 video. Download. The video doesn't render in the dev server, and the fallback text provides no context for what the video shows. This leaves a noticeable blank space in the guide.

Fix: Add an alt-text description of what the video demonstrates, or a static screenshot with caption as fallback.

6. Step 4 ("Customize your workflow") is optional but critically important

Step 4 teaches users to edit .github/workflows/daily-repo-status.md, run gh aw compile, and push. But it's marked "(optional)". For new users, gh aw compile is the first time they see a key command — and it appears in an optional step. If they skip Step 4, they never learn that editing the markdown requires recompilation.

Fix: Move the gh aw compile concept to a dedicated "How it works" callout, or make Step 4 non-optional with a brief note.

7. CLI Commands page has an overwhelming command list with no grouping by experience level

The CLI page lists 30+ commands in a flat table. There's no "Start here" section that highlights the 4-5 commands a beginner needs (init, add-wizard, compile, run). The "Most Common Commands" table at top helps, but it's just a table with no guidance on order or context.

Fix: Add a "Beginner path" section or numbered steps at the top of the CLI page.

8. Search is broken in dev mode

The search bar shows: "Search is only available in production builds. Try building and previewing the site to test it out locally." This is expected for Astro/Starlight, but a new contributor testing the docs locally will find this confusing and might think the docs are broken.

Fix: This is acceptable behavior — just add a comment in the README/CONTRIBUTING noting that search is unavailable in dev mode.

---

🟢 What Worked Well

• Homepage hero copy is excellent — "Repository automation, running the coding agents you know and love, with strong guardrails in GitHub Actions" is clear and compelling on first read.
• Security architecture section is a genuine differentiator — the five security layers are explained clearly and the Mermaid diagram renders well.
• Prerequisites list in Quick Start is thorough — listing OS, gh CLI version, AI account, and GitHub Actions toggle is exactly what beginners need.
• add-wizard interactive flow is well-described — the bullet list of what the wizard does (check prerequisites → select engine → set secret → add workflow → trigger run) is a clear mental model.
• "Estimated time: 10 minutes" at the top of Quick Start sets expectations perfectly.
• The navigation sidebar is comprehensive — covers Introduction, Setup, Guides, Patterns, Reference, and Troubleshooting with good hierarchy.
• Early-development warning callout — the honest "⚠️ early development, use with caution" note builds trust rather than undermining it.

---

Recommendations

Quick wins (high impact, low effort):

1. 🔗 Add a direct link to the githubnext/agentics repo in Step 2, with a note on visibility
2. 📝 Add an inline "How to create COPILOT_GITHUB_TOKEN" summary (3 steps) in Step 2
3. 💡 Define "frontmatter" with a callout box on first use in Step 4
4. 🎬 Add a static screenshot fallback for the video in Quick Start

Medium-term improvements:
5. 🧭 Rename "Peli's Agent Factory" in nav to something universally understood (e.g., "Workflow Gallery")
6. 📊 Add a "Beginner path" section to the CLI Commands page with 5 key commands
7. 🔑 Make gh aw compile visible earlier in the guide, not buried in an optional step

Longer-term:
8. 📖 Add a "Concepts" page explaining key terms: frontmatter, lock file, safe outputs, agent engine — linked from the Quick Start
9. 🗺️ Add a "Learning path" visual on the homepage: Install → Add Workflow → Customize → Create Your Own

---

References:

• §24837327233

Generated by Documentation Noob Tester · ● 2.1M · ◷

• expires on Apr 24, 2026, 1:28 PM UTC

[github-actions[bot]]

This discussion has been marked as outdated by Documentation Noob Tester.

A newer discussion is available at Discussion #28279.