Skip to content

Add evidence-based repository audits and evals - #2

Draft
study8677 wants to merge 1 commit into
mainfrom
agent/repository-audit-and-evals
Draft

Add evidence-based repository audits and evals#2
study8677 wants to merge 1 commit into
mainfrom
agent/repository-audit-and-evals

Conversation

@study8677

Copy link
Copy Markdown
Owner

What changed

  • add an inspect-first repository architecture audit mode across the native skill, Codex AGENTS.md, and Cursor rule
  • classify findings as observed, inferred, or unknown; rank a high-leverage Top 3 with validation and rollback paths
  • bundle a read-only repository inventory tool plus architecture-review and ADR output templates
  • add native Codex metadata and make native skill installation the recommended Codex path
  • add 14 behavior evaluation cases, a scoring rubric, deterministic cross-surface/reference checks, unit tests, and GitHub Actions CI

Why

The project already had strong architecture knowledge, but its highest-leverage gap was turning that knowledge into a repeatable daily workflow for existing codebases while preventing prompt and platform variants from silently drifting. This change adds both the user-visible repository audit workflow and the feedback loop needed to evolve it safely.

Validation

  • python3 scripts/check_quality.py
  • PYTHONDONTWRITEBYTECODE=1 python3 -m unittest discover -s tests -v (3/3 passed)
  • quick_validate.py skills/architecture-copilot
  • native-skill copy/install smoke test
  • local Markdown-link and YAML parsing checks
  • blind forward test: repository audit read the new reference, ran the bundled inventory, cited real paths, separated evidence levels, and produced a read-only Top 3
  • blind non-trigger test: a FastAPI unit-test request correctly stayed out of architecture mode

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant