Skip to content

Add live provider test contracts#33

Draft
mavam wants to merge 2 commits into
mainfrom
topic/better-testing
Draft

Add live provider test contracts#33
mavam wants to merge 2 commits into
mainfrom
topic/better-testing

Conversation

@mavam

@mavam mavam commented Jun 13, 2026

Copy link
Copy Markdown
Owner

🔍 Problem

  • Provider tests covered mocked behavior, but there was no CI path for validating real API-backed providers with GitHub Secrets.
  • The default test suite still needs to stay fast, deterministic, and credential-free.

🛠️ Solution

  • Add opt-in live API contracts for every API-backed provider capability, using env-var credentials and stable shape/content assertions.
  • Split package scripts so npm test stays offline and npm run test:live runs selected live contracts sequentially.
  • Add a separate secret-backed GitHub Actions live API job for manual, scheduled, and same-repository PR runs.
  • Document local and GitHub Secret-backed live testing, including provider and capability filters.

💬 Review

  • Focus on the provider/capability contract matrix, the missing-secret failure mode, and the CI trigger/fork behavior.
  • The live tests intentionally avoid exact-output snapshots so provider ranking and generated text drift do not cause brittle failures.

mavam added 2 commits June 13, 2026 11:38
Add an opt-in live API test suite that exercises provider routing with real credentials from environment variables. Keep the default test path offline and document how CI and local runs select providers and capabilities.

Assisted-by: GPT-5 (Codex)
Document the new opt-in live provider testing workflow in the unreleased changelog so the PR carries release notes for maintainers.

Assisted-by: GPT-5 (Codex)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant