feat: Model-selection policy schema for agentic workflows

[lpcox]

Summary

Define a model-selection policy object that captures how agentic workflows choose, validate, and fall back between models at runtime. The schema lives in this repo (gh-aw-firewall) alongside other policy primitives (network ACLs, seccomp profiles), but is understood by the gh-aw compiler and enforced by AWF at execution time.

Companion to gh-aw#29191 (model fallback feature request).

Proposed Policy Schema

{
  "$schema": "https://github.com/github/gh-aw-firewall/schemas/model-policy.v1.json",
  "version": "1",

  // Primary model selection
  "model": {
    "id": "gpt-5.2",                    // requested model
    "reasoning_effort": "medium",        // optional engine-specific param
    "provider": "copilot"                // copilot | anthropic | openai | custom
  },

  // Fallback chain — tried in order when primary is unavailable
  "fallback": [
    { "id": "gpt-4.1", "provider": "copilot" },
    { "id": "claude-sonnet-4-20250514", "provider": "anthropic" },
    { "strategy": "auto" }              // sentinel: pick best available
  ],

  // Constraints applied to ALL model selections (primary + fallbacks)
  "constraints": {
    "capabilities": ["tool-use", "vision"],  // required capabilities
    "max_context_window": null,              // null = no limit
    "min_context_window": 128000,            // minimum tokens
    "cost_tier": "standard"                  // standard | premium | economy
  },

  // Behavior when no model satisfies constraints
  "on_unavailable": "fail",  // "fail" | "warn-and-use-best" | "queue"

  // Audit/observability
  "audit": {
    "log_selection": true,          // log which model was selected and why
    "log_fallback_reason": true     // log why primary was skipped
  }
}

Integration Points

1. Compiler (gh-aw compile)

The compiler reads model-selection policy from workflow frontmatter and serializes it into the lock file:

# In .md workflow frontmatter
model: gpt-5.2
model-policy:
  fallback: [gpt-4.1, claude-sonnet-4-20250514, auto]
  constraints:
    capabilities: [tool-use]
    min_context_window: 128000
  on_unavailable: fail

The compiler validates:

• Model IDs against a known registry (warn on unknown, don't block)
• Constraint fields are well-formed
• Fallback chain doesn't exceed max depth (e.g., 5)

2. AWF Runtime Enforcement

At container startup, AWF:

1. Reads the model policy from the workflow metadata (passed via env var AWF_MODEL_POLICY_B64 or similar)
2. Queries available models via the API proxy sidecar (GET /models)
3. Resolves the effective model by walking: primary → fallback[0] → fallback[1] → ... → auto
4. Applies constraints to filter candidates (capabilities, context window, cost tier)
5. Sets AWF_RESOLVED_MODEL env var in the agent container
6. Emits audit log entries for observability

3. API Proxy Sidecar

The api-proxy can enforce the resolved model:

• If agent requests a model different from AWF_RESOLVED_MODEL, either:
  • Rewrite the request to use the resolved model (transparent enforcement)
  • Reject with 400 and guidance (strict enforcement)
• Configuration: "enforcement": "rewrite" | "reject" | "passthrough"

Schema Location

gh-aw-firewall/
├── schemas/
│   └── model-policy.v1.json          # JSON Schema definition
├── src/
│   ├── model-policy.ts               # Parser + validator
│   └── model-resolver.ts             # Resolution logic (primary → fallback → auto)
└── docs/
    └── model-selection-policy.md      # Specification document

Design Principles

1. Declarative over imperative — Policy describes intent, not implementation
2. Fail-safe defaults — Without explicit policy, current behavior (hard fail) is preserved
3. Auditable — Every model selection decision is logged with reasoning
4. Composable — Org-level policies can constrain repo-level policies (future: org policy inheritance)
5. Engine-agnostic — Works across Copilot, Claude, Codex, and custom engines

Open Questions

• Should auto strategy use capability matching or just pick the "best" available model?
• How does org-level model governance interact? (e.g., org bans certain models)
• Should the policy support time-based selection? (e.g., use cheaper model for scheduled jobs)
• Is AWF_MODEL_POLICY_B64 the right transport, or should it be a file mount?

Related

• gh-aw#29191 — Model fallback feature request (upstream)
• sweagentd#11264 — CCA jobs failing with model_not_supported (motivating incident)
• Network policy precedent: --allow-domains → Squid ACL (same pattern: declarative policy → runtime enforcement)

[lpcox]

Feedback from PR #2310 Review

@pelikhan commented:

I think we want a policy that support multiple models to support sub agents

Proposed Enhancement: Multi-Model Policy for Sub-Agents

The current schema defines a single model + fallback chain, which works for a workflow with one agent. However, orchestrator workflows may dispatch sub-agents that need different models (e.g., a GPT-5.2 orchestrator delegating to Claude sub-agents for specific tasks, or using a cheaper model for simple classification sub-tasks).

Proposed extension — add an optional agents map that defines per-agent model policies:

{
  "version": "1",

  // Default model policy (applies to the primary agent)
  "model": { "id": "gpt-5.2", "provider": "copilot" },
  "fallback": [{ "id": "gpt-4.1", "provider": "copilot" }],

  // Per-agent overrides for sub-agents
  "agents": {
    "classifier": {
      "model": { "id": "gpt-4.1-mini", "provider": "copilot" },
      "constraints": { "cost_tier": "economy" }
    },
    "researcher": {
      "model": { "id": "claude-sonnet-4-20250514", "provider": "anthropic" },
      "fallback": [{ "strategy": "auto" }],
      "constraints": { "capabilities": ["tool-use", "vision"] }
    }
  },

  // Global constraints still apply as a floor
  "constraints": { "capabilities": ["tool-use"], "min_context_window": 128000 },
  "on_unavailable": "fail",
  "audit": { "log_selection": true, "log_fallback_reason": true }
}

Key considerations:

• Each entry in agents inherits from the top-level policy but can override model, fallback, constraints
• Agent names correspond to sub-agent identifiers in the workflow (TBD how these map in gh-aw frontmatter)
• AWF resolves each agent's model independently at startup
• The auto sentinel in a sub-agent's fallback still respects the global constraints as a floor
• Fallback depth limit (5) applies per-agent, not globally

[lpcox]

Additional Feedback: Model Categories / Capability Classes

Rather than requiring workflows to specify exact model IDs (which change frequently and vary by provider), the policy should support category-based model selection using semantic labels like "opus", "sonnet", "mini", "reasoning", etc.

Problem

Exact model IDs are brittle:

• claude-sonnet-4-20250514 → gets superseded by claude-sonnet-4-20250715
• gpt-5.2 → may not be available in all regions/plans
• Workflow authors shouldn't need to track model release cycles

Proposed: category Field

Allow models to be specified by category instead of (or in addition to) exact ID:

{
  "version": "1",

  // Category-based selection (preferred for most workflows)
  "model": { "category": "sonnet", "provider": "anthropic" },

  // Sub-agent with reasoning category
  "agents": {
    "planner": { "model": { "category": "reasoning", "provider": "copilot" } },
    "worker":  { "model": { "category": "mini", "provider": "copilot" } }
  },

  "fallback": [
    { "category": "sonnet" },          // any sonnet-class model
    { "category": "standard" },        // any standard-tier model
    { "strategy": "auto" }
  ]
}

Category Taxonomy

Category	Semantics	Example Models
opus	Highest capability, premium cost	claude-opus-4, gpt-5.2
sonnet	High capability, standard cost	claude-sonnet-4, gpt-4.1
mini	Fast/cheap, lower capability	gpt-5.4-mini, claude-haiku-4
reasoning	Extended thinking / chain-of-thought	o3, claude-opus-4 (extended thinking)
standard	General-purpose, balanced cost/quality	Any model at standard tier
premium	Best available regardless of cost	Any model at premium tier
economy	Cheapest available meeting constraints	Any model at economy tier
vision	Multimodal with image understanding	Models with vision capability
code	Optimized for code generation	codex, code-specialized models

Resolution Algorithm

When AWF resolves a category-based model spec:

1. Query available models from the provider (via /models endpoint or registry)
2. Filter by category — each model in the registry has one or more category tags
3. Apply constraints — filter remaining candidates by capabilities, min_context_window, cost_tier
4. Rank candidates — prefer: (a) newest version, (b) highest capability score, (c) lowest cost within tier
5. Select top candidate — resolve to concrete model ID for the session

Category Registry

The category→model mapping lives in the AWF model registry (updatable without redeploying):

// model-registry.json (maintained by gh-aw-firewall)
{
  "models": [
    {
      "id": "claude-sonnet-4-20250514",
      "provider": "anthropic",
      "categories": ["sonnet", "standard", "code"],
      "capabilities": ["tool-use", "vision"],
      "context_window": 200000,
      "cost_tier": "standard",
      "released": "2025-05-14"
    },
    {
      "id": "gpt-5.4-mini",
      "provider": "copilot",
      "categories": ["mini", "economy"],
      "capabilities": ["tool-use"],
      "context_window": 128000,
      "cost_tier": "economy",
      "released": "2026-03-01"
    }
  ]
}

Precedence Rules

• id takes precedence over category if both are specified (category used only for fallback)
• Multiple categories can be combined: { "category": ["reasoning", "vision"] } = must match ALL
• Category resolution happens at runtime (not compile time) so new models are automatically picked up
• Unknown categories are treated as errors at compile time (validated against known taxonomy)

Benefits

• Future-proof: New models automatically become candidates without workflow changes
• Intent-driven: Workflow authors express what they need not which specific model
• Sub-agent friendly: Easy to say "use a cheap model for classification, a reasoning model for planning"
• Provider-agnostic: { "category": "sonnet" } could resolve to Claude or an equivalent from another provider

[lpcox]

@pelikhan updated the issue

[pelikhan]

Litellm maintains a registry of models, in practice it's an ever changing set of entries. It's work to maintain but maybe agents can do that automatically.

I'd rather keep this minimalistic: no release data , no window size, just mappings between names for starter.