[prompt-analysis] Copilot PR Prompt Analysis - 2026-08-21

[github-actions[bot]]

🤖 Copilot PR Prompt Pattern Analysis - 2026-08-21

Summary

Analysis Period: Last 30 days
Total PRs: 1000 | Merged: 767 (76.7%) | Closed: 211 (21.1%) | Open: 22 (2.2%)
Completed merge rate (merged / (merged+closed)): 78.4%

Prompt Categories and Success Rates

Category	Total	Merged	Closed	Success Rate
Refactoring	22	20	2	90.9%
Feature Addition	256	206	44	82.4%
Documentation	94	73	16	82.0%
Other	31	23	7	76.7%
Bug Fix	395	297	91	76.5%
Testing	202	148	51	74.4%

(Categorized by keyword matching on title + body; bug-fix is the largest and most common category.)

Prompt Analysis

✅ Successful Prompt Patterns

Common characteristics in merged PRs:

• Average prompt length: 168 words (notably shorter than closed PRs)
• Most common keywords: workflow, github, changes, added, existing, lock, coverage, agent, yaml, path
• Prompts tend to reference specific files, packages, or symbols (e.g., pkg/colorwriter, purity_scan.go) rather than broad goals

Example successful prompts:

1. PR Provision a sandbox-usable CPython for chart-generating workflows #54519: "Provision a sandbox-usable CPython for chart-generating workflows" → Merged
2. PR Align "MCP server" capitalization to "MCP Server" in install step #54512: "Align "MCP server" capitalization to "MCP Server" in install step" → Merged
3. PR Simplify purity_scan.go: dedupe Uses/Defs fallback and local recording #54500: "Simplify purity_scan.go: dedupe Uses/Defs fallback and local recording" → Merged

❌ Unsuccessful Prompt Patterns

Common characteristics in closed PRs:

• Average prompt length: 230 words (~37% longer than merged prompts)
• Most common keywords: workflow, github, actions, generated, copilot, engine, repo, keep, call
• Higher incidence of [WIP]-tagged or exploratory titles signaling unfinished/uncertain scope
• Broader, less targeted asks (e.g., "detect Squad workflows", "split file by concern") without a narrow, verifiable acceptance criterion

Example unsuccessful prompts:

1. PR Prevent placeholder PR description updates #54494: "Prevent placeholder PR description updates" → Closed
2. PR [WIP] Fix failed jobs in PR Description Updater #54493: "[WIP] Fix failed jobs in PR Description Updater" → Closed
3. PR Enable detection for Squad workflows #54399: "Enable detection for Squad workflows" → Closed

Key Insights

• Pattern 1: Refactoring prompts have the highest success rate (90.9%), likely because they target well-defined, mechanically verifiable changes (splitting a file, renaming, deduping) with clear success criteria.
• Pattern 2: Shorter prompts correlate with higher success — merged PR prompts average 168 words vs. 230 for closed PRs, suggesting overly long/ambiguous task descriptions increase the chance of scope drift or rejection.
• Pattern 3: Testing-category prompts have the lowest success rate (74.4%) among substantive categories, and [WIP] labeling appears disproportionately in closed PRs, indicating unfinished or exploratory asks are more likely to be abandoned/closed.

Recommendations

1. DO: Reference specific files, functions, or packages in the prompt (e.g., pkg/colorwriter/README.md) — this correlates with merged outcomes.
2. DO: Keep prompts concise and scoped to a single, verifiable change; aim for well under 200 words.
3. AVOID: Marking or starting tasks as [WIP] / exploratory — these have a much higher closure rate; only submit prompts once scope is fully defined.

Historical Trends

Date	PRs	Merged	Closed	Success Rate	Top Category
2026-08-21	1000	767	211	78.4%	bug_fix
2026-07-06	1000	793	193	80.4%	test
2026-07-05	1000	805	191	80.8%	bug_fix
2026-07-04	1000	798	186	81.1%	bug_fix
2026-07-03	1000	801	178	81.8%	bug_fix

Trend: Success rate has declined gradually from ~82% (early July) to 78.4% (current), a ~3.4pp drop over the observed period. Bug fixes remain the dominant category by volume, though refactoring and feature prompts now show higher success rates than bug fixes — suggesting bug-fix prompts may benefit from tighter, more specific framing (error messages, reproduction steps, target files) to recover success rate.

---

Generated by Copilot PR Prompt Analysis (Run: 32525062830)

Generated by 🔍 Copilot PR Prompt Pattern Analysis · auto · 21.6 AIC · ⌖ 5.62 AIC · ⊞ 11.1K · ◷

• expires on Aug 22, 2026, 12:50 PM UTC-08:00

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-08-22T20:50:34.718Z.

Closed by Workflow