[docs-noob-tester] 📚 Documentation Noob Test Report - 2026-04-27

[github-actions[bot]]

Summary

• Date of test: 2026-04-27
• Pages visited:
  • https://github.github.com/gh-aw/ (Home)
  • https://github.github.com/gh-aw/setup/quick-start/ (Quick Start)
  • https://github.github.com/gh-aw/setup/cli/ (CLI Commands)
• Overall impression: As a complete beginner, the home page is visually clear and inspiring — I immediately understand what the tool does. However, the Quick Start guide assumes some familiarity with concepts like "frontmatter", GitHub Actions, and CLI tooling that could trip up true newcomers.

---

🔴 Critical Issues Found

1. Term "frontmatter" used without definition

In Quick Start Step 4, the guide refers to "frontmatter (the YAML configuration block between --- markers)" but this explanation is tucked inside a parenthetical. A beginner may have already been confused several steps earlier without knowing what frontmatter is. It should be defined upfront or linked to a dedicated explainer.

2. add-wizard command format is non-obvious

Step 2 uses:

gh aw add-wizard githubnext/agentics/daily-repo-status

The format <owner>/<repo>/<workflow-name> is explained right below, but the command itself looks cryptic to a newcomer who doesn't know what "agentics" is or why there are 3 slash-separated parts. A brief sentence before the command would help: "This installs a pre-built sample workflow from the githubnext/agentics community repository."

3. Secret setup is complex and fragmented

Step 2 mentions setting up one of four different secrets (COPILOT_GITHUB_TOKEN, ANTHROPIC_API_KEY, OPENAI_API_KEY, GEMINI_API_KEY) but only links to the Authentication reference page. For a beginner following a "Quick Start", this is a potential blocker. A beginner doesn't know which secret they need until they pick an engine, and the engine selection happens inside the interactive wizard (which they haven't run yet). The circular dependency is confusing.

---

🟡 Confusing Areas

1. Home page tagline uses jargon

"Repository automation, running the coding agents you know and love, with strong guardrails in GitHub Actions."

A brand-new user may not know what "coding agents" means, or what "guardrails" implies in this context. A simpler version could be: "Automate your GitHub repository using AI — safely, with one markdown file."

2. Early warning is very prominent but vague

The "i Note: GitHub Agentic Workflows is in early development..." block is shown prominently on both the home page and Quick Start guide. The phrase "things can still go wrong" without specific examples is anxiety-inducing for beginners. What kinds of things? Can it delete my code? A single concrete example of what the risk is (e.g., "the AI might create issues or PRs that need human review") would be reassuring rather than alarming.

3. Quick Start step titles are inconsistent

Steps 1–4 are numbered sequentially, but Step 3 ("Wait for the workflow to complete") is passive — the user does nothing. It reads as a heading without a corresponding action. Could be clarified as: "⏳ Wait for results (2–3 minutes)".

4. "What's next?" points to a blog post as primary resource

The Quick Start ends with a link to Peli's Agent Factory blog post as a primary "what's next" resource. This is an odd choice for beginners — a blog post feels informal and harder to navigate than structured documentation. The creating-workflows guide and overview links are listed below it but deserve to be more prominent.

5. CLI Commands page is very long

The CLI Commands page is ~36KB of content. For a new user looking for "how do I run a workflow", the most important commands (gh aw run, gh aw compile) are easy to miss in the wall of commands. The "Most Common Commands" table at the top is helpful, but the rest of the page needs better visual hierarchy or collapsible sections.

---

🟢 What Worked Well

• Home page is clear and welcoming — The hero section immediately communicates what the product does, and the two CTAs ("Quick Start with CLI" and "Creating Workflows") are prominent.
• Prerequisites are well-listed — The Quick Start has a clean, checkboxed prerequisites section (AI Account, GitHub Repository, GitHub Actions enabled, GitHub CLI, OS requirement). This is great for newcomers.
• Video tutorial on Quick Start — The embedded video "Install the extension, add a workflow, and trigger a run from the CLI" is an excellent addition for visual learners.
• Security section on home page is excellent — The "Guardrails Built-In" section with the Mermaid diagram clearly explains the security model. This is reassuring to skeptical users.
• Estimated time (⏱️ 10 minutes) — Setting expectations upfront is a great UX practice.
• TIP callouts — The > [!TIP] blocks in the Quick Start (auth issues fallback, troubleshooting links) are well-placed.
• Most Common Commands table — The CLI page starts with a summary table of the most-used commands, which is exactly right for beginners scanning for the essentials.

---

Recommendations

Quick wins (high impact, low effort):

1. Add a one-sentence definition of "frontmatter" the first time it appears in Quick Start.
2. Add a brief intro sentence before the add-wizard command explaining what it installs.
3. Move secret setup guidance earlier (before the add-wizard step) or add an inline summary: "You'll need an API key for your chosen AI engine — see setup instructions."
4. Replace "things can still go wrong" in the warning note with a concrete, brief example.

Longer-term improvements:
5. Add a "Concepts" or "Glossary" page explaining: frontmatter, workflow, engine, safe outputs, lock file — terms used throughout but never defined for beginners.
6. Consider adding a "Which AI engine should I use?" decision helper to the Quick Start.
7. Add collapsible sections or a table of contents sidebar entry for the CLI Commands page to make it easier to navigate.
8. Make the "What's next?" section on Quick Start point to structured docs first, with the blog post as optional exploration.

---

Screenshots

📎 [home-page.png] — Home page showing hero section and CTAs:

---

References:

• §24997242366

Generated by Documentation Noob Tester · ● 3.2M · ◷

• expires on Apr 28, 2026, 1:26 PM UTC

[github-actions[bot]]

This discussion has been marked as outdated by Documentation Noob Tester.

A newer discussion is available at Discussion #28950.