[docs-noob-tester] 📚 Documentation Noob Test Report - 2026-05-03

[github-actions[bot]]

Summary

• Date: 2026-05-03
• Pages analyzed: Home (/gh-aw/), Quick Start (/gh-aw/setup/quick-start/), CLI Commands (/gh-aw/setup/cli/)
• Overall impression: The docs are generally well-organized and have a clear narrative. However, as a complete beginner, I hit several stumbling blocks — particularly around unexplained jargon ("frontmatter", "lock file", "compilation"), a missing explicit "what is this tool" section on the home page, and a quick start that jumps from zero to gh aw add-wizard without explaining what happens under the hood.

⚠️ Note on screenshots: The documentation preview server could not bind to the network due to sandbox restrictions in this test run. Analysis is based on source markdown files directly.

---

🔴 Critical Issues

1. Home page doesn't explain what the tool is in simple terms

The hero tagline — "Repository automation, running the coding agents you know and love, with strong guardrails in GitHub Actions" — is packed with jargon for a brand-new user. What's a "coding agent"? What are "guardrails"? What is "GitHub Actions"?

A beginner landing on the home page needs a plain one-sentence answer to "What does this install on my computer / my repo?" before seeing a feature grid. The current intro paragraphs are great but appear below the fold below the feature grid.

Recommendation: Add a single-sentence plain-English "elevator pitch" at the very top, e.g.: "GitHub Agentic Workflows lets you write simple text files that tell an AI agent what to do in your GitHub repository — automatically, on a schedule or when events happen."

---

2. Quick Start uses "frontmatter" without a definition

The Quick Start (Step 4) says:

"If you changed the frontmatter (the configuration block between the --- markers...)"

The first time "frontmatter" appears, it's in parentheses with a partial definition, but the term is used again in Step 2 without explanation. A complete beginner writing their first markdown file won't know this is YAML or that the compiler cares about it. There is a link to /gh-aw/reference/frontmatter/ but it's buried.

Recommendation: Bold the first mention and link it immediately: "Edit the frontmatter — the YAML block between --- markers at the top of the file."

---

3. "Lock file" concept is introduced without upfront explanation

The Quick Start mentions .lock.yml in Step 2 without ever defining it. The Glossary has a good definition but there's no breadcrumb from the Quick Start. A beginner will wonder: "Why are there two files? Which one do I edit? Which one does GitHub run?"

Recommendation: Add a callout box at the top of the Quick Start:

💡 Two files, one workflow: Every workflow has a .md file (you edit this) and a .lock.yml file (auto-generated — don't edit this). GitHub Actions runs the lock file; the CLI keeps it in sync.

---

4. COPILOT_GITHUB_TOKEN secret setup is unclear

In Quick Start Step 2, setting up the required secret is mentioned as item 3 of the add-wizard interactive steps: "COPILOT_GITHUB_TOKEN (a separate GitHub token with Copilot access — distinct from the default GITHUB_TOKEN)". But a new user will not know:

• How to create this token
• Where exactly to put it (repository secrets? organization secrets?)
• What permissions it needs

The wizard likely handles this interactively, but the docs don't reassure the reader. If the wizard fails or the user wants to do this manually, they have to hunt through the Auth reference.

Recommendation: Add a one-line note: "The wizard will guide you through creating and setting this secret. If you prefer to do it manually, see Authentication."

---

🟡 Confusing Areas

5. "Add-wizard" vs "add" vs "new" — when do I use which?

The CLI Commands page lists three commands for getting workflows: add-wizard, add, and new. For a beginner these are confusingly similar:

• add-wizard — interactive setup for an existing community workflow
• add — non-interactive version of the same
• new — create a workflow from scratch

This distinction is buried. A beginner will not know whether to run gh aw add-wizard or gh aw new for their first workflow.

Recommendation: Add a "Which command should I use?" decision tree at the top of the Commands section, or a brief comparison table like:

I want to...	Command
Add a pre-built workflow interactively	add-wizard
Add a pre-built workflow in a script	add
Write my own workflow	new

---

6. The home page doesn't have a visible "Get Started" path for the most common scenario

The hero has two CTA buttons: "Quick Start with CLI" and "Creating Workflows". The distinction is unclear — both seem like "getting started". The "Quick Start with CLI" label implies I need to use a CLI, which may deter users who just want to understand the tool first.

Recommendation: Rename to something like "Install & Try It (10 min)" to set concrete expectations.

---

7. CLI page is reference-heavy, not task-oriented

The CLI Commands page leads with a dense reference table and immediately dives into installation variants (pinning versions, standalone installer, GitHub Actions setup action, GitHub Enterprise). A brand-new user just wants to know: "Install gh, then run this one command." The enterprise/GHE section is particularly out of place at the top.

Recommendation: Restructure the CLI page with a clear "Most Users" path first, and collapse GHE/advanced sections into a <details> block or move them to a dedicated page.

---

8. No prerequisite for understanding GitHub Actions

The Quick Start lists GitHub Actions as a prerequisite with a checkbox ("GitHub Actions enabled") but doesn't explain what GitHub Actions is or link to an intro. A user who has never used GitHub Actions won't know what "workflows running in GitHub Actions" means.

The Introduction section has a GitHub Actions Primer — but this is not prominently linked from the Quick Start.

Recommendation: Add a note: "New to GitHub Actions? Read our 5-minute primer first."

---

🟢 What Worked Well

1. 10-minute estimate up front — The Quick Start explicitly says "⏱️ Estimated time: 10 minutes". This sets expectations perfectly for a new user.

2. The security explanation on the home page is excellent — The five security layers (read-only tokens, zero secrets in agent, firewall, safe outputs, threat detection) with the Mermaid diagram is the clearest and most reassuring part of the docs. A beginner will immediately understand why this tool is safe to use.

3. The example workflow is concrete and relatable — The "Daily Issues Report" example on the home page with the full markdown snippet is very helpful. Seeing real code alongside plain English makes the concept click.

4. Step 4 (Customize) includes good [!NOTE] callout — The note explaining the two-part structure of a workflow file (frontmatter vs body) is excellent. More of this pattern would help throughout the Quick Start.

5. Prerequisite checklist format — Using a bullet list with checkboxes for prerequisites in the Quick Start is a good pattern. It's easy to scan and actionable.

6. add-wizard is smart UX — Having a guided wizard that handles secrets, engine selection, and first run in one command is a strong choice for a quick start. The docs correctly make this the primary entry point.

---

Recommendations (Prioritized)

Quick wins (low effort, high impact)

1. Add a plain-English elevator pitch above the feature grid on the home page (~1 sentence)
2. Link "frontmatter" on first use in the Quick Start with an inline definition
3. Add a "two files" callout box at the top of Quick Start explaining .md vs .lock.yml
4. Add a "Which command should I use?" mini-table at the top of the CLI page Commands section
5. Link the GitHub Actions Primer from the Quick Start prerequisites section

Medium-term improvements

6. Restructure the CLI page — Move GHE/enterprise sections to a collapsible or separate page; lead with the happy path
7. Rename the hero CTA from "Quick Start with CLI" to something that sets time and outcome expectations
8. Add a "What happens next" section after Step 3 of Quick Start (what does the issue look like? what if the run fails?)

Longer-term improvements

9. Add an interactive "What kind of user are you?" landing section: repository owner, developer, team admin
10. Glossary sidebar widget: Link key terms (frontmatter, lock file, safe outputs, engine) inline throughout the docs with hover definitions
11. Add a troubleshooting mini-section to the Quick Start itself for the most common first-run failures (wrong secret name, token permissions, GitHub Actions not enabled)

References: §25270251150

Warning

Firewall blocked 5 domains

The following domains were blocked by the firewall during workflow execution:

• accounts.google.com
• android.clients.google.com
• clients2.google.com
• safebrowsingohttpgateway.googleapis.com
• www.google.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "accounts.google.com"
    - "android.clients.google.com"
    - "clients2.google.com"
    - "safebrowsingohttpgateway.googleapis.com"
    - "www.google.com"

See Network Configuration for more information.

Generated by Documentation Noob Tester · ● 2.5M · ◷

• expires on May 4, 2026, 5:02 AM UTC

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-05-04T05:02:21.648Z.

Closed by Workflow