[WIP] feat: add skill evaluation framework

[abhishekgarg18]

Add comprehensive skill evaluation system following agentskills.io patterns.

Features

• Automated testing for Cloud Service and 6.5 LTS dispatcher skills
• Local execution with npm scripts for rapid iteration
• CI/CD integration via GitHub Actions on PRs
• LLM-based grading using Claude API for assertion validation
• Variant-aware testing with separate test cases for each AEM flavor

Components

Test Cases

• Cloud Service dispatcher config-authoring (3 test cases)
• 6.5 LTS dispatcher config-authoring (3 test cases)
• Assertions covering filters, vhosts, cache, security

Eval Runner

• Node.js-based evaluation engine
• Sequential test execution with timing metrics
• Haiku grader for cost-efficient assertion validation
• Results saved to evals-workspace/ (git-ignored)

GitHub Actions

• Triggered on PR when dispatcher skills change
• Path-based filtering for targeted execution
• Posts detailed results as PR comments
• Uploads full results as artifacts (30-day retention)

Documentation

• README.md: comprehensive guide (writing evals, local testing, CI/CD)
• QUICKSTART.md: 5-minute getting started
• ARCHITECTURE.md: technical deep-dive and data flow

Usage

Local Testing

cd evals/runner
npm install
export ANTHROPIC_API_KEY=sk-ant-...
npm run eval -- --skill cloud-service/dispatcher/config-authoring

CI/CD

Automatically runs on PRs when:

• skills/aem/cloud-service/skills/dispatcher/** changes
• skills/aem/6.5-lts/skills/dispatcher/** changes
• evals/skills/** changes

Requires: ANTHROPIC_API_KEY secret in repository settings

Files Added

• evals/skills/cloud-service/dispatcher/config-authoring/evals.json
• evals/skills/6.5-lts/dispatcher/config-authoring/evals.json
• evals/runner/{package.json,eval-runner.js,.npmrc}
• .github/workflows/eval-skills.yml
• .github/scripts/generate-eval-report.js
• evals/{README.md,QUICKSTART.md,ARCHITECTURE.md}

Files Modified

• package.json: added eval scripts
• .gitignore: exclude evals-workspace/

Description

Related Issue

Motivation and Context

How Has This Been Tested?

Screenshots (if appropriate):

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

• I have signed the Adobe Open Source CLA.
• My code follows the code style of this project.
• My change requires a change to the documentation.
• I have updated the documentation accordingly.
• I have read the CONTRIBUTING document.
• I have added tests to cover my changes.
• All new and existing tests passed.