feat(operator): operator UI, GitHub PAT auth, AI rule suggester

[cbullinger]

Summary

Builds out the operator UI into a production-ready diagnostic + self-service tool, gated by GitHub PAT authentication with repo-derived roles, and adds an AI rule suggester backed by the hosted Anthropic API through the Grove Foundry APIM gateway.

Commits on feat/operator-ui-audit → main:

SHA	What
8f39997	Comprehensive operator UI with writer-facing features
fe927fd	Optional GitHub PAT auth with role + per-repo access checks
80c5da9	Remove token auth mode, require GitHub PAT + auth repo
8c1a76a	AI rule suggester with full UI-based Ollama management
2735584	gosec G107/G704 SSRF hardening on the GitHub permission check
3551a0b	Show operator UI + AI settings in the startup banner
c8508e5	Wire operator UI, enable audit, populate changelog
473bdb3	Restore v0.3.0 heading in CHANGELOG
62b6f33	Anthropic LLM provider for AI rule suggester
b75df04	Route Anthropic client through Grove Foundry APIM gateway
579edb9	cmd/test-llm smoke-test CLI

Configuration

Env var	Required	Description
OPERATOR_UI_ENABLED	for operator UI	true to mount /operator/ routes
OPERATOR_AUTH_REPO	when UI enabled	owner/repo — the user's permission on this repo determines their role
OPERATOR_REPO_SLUG	optional	For clickable GitHub links in audit/trace rows
OPERATOR_RELEASE_GITHUB_TOKEN	optional	PAT with contents:write to create release tags from the UI
LLM_PROVIDER	optional	anthropic (default in prod) or ollama (local)
LLM_BASE_URL	optional	Per-provider default. For the Grove Foundry gateway: https://grove-gateway-prod.azure-api.net/grove-foundry-prod/anthropic
LLM_MODEL	optional	Per-provider default. claude-haiku-4-5 for the gateway, qwen2.5-coder:7b for Ollama
ANTHROPIC_API_KEY	for anthropic (dev)	Direct value; overrides Secret Manager if set
ANTHROPIC_API_KEY_SECRET_NAME	for anthropic (prod)	Secret Manager resource to load on boot; used in Cloud Run

Startup fails if OPERATOR_UI_ENABLED=true but OPERATOR_AUTH_REPO is missing — prevents an accidentally-open operator UI. Missing ANTHROPIC_API_KEY is non-fatal: the UI shows "not configured" and every other feature keeps working.

Authentication & authorization

Each user authenticates with their personal GitHub PAT. Their permission on OPERATOR_AUTH_REPO determines their role:

GitHub permission	Operator UI role	Can do
admin / maintain	operator	Everything: view, replay, release, AI settings
write / triage / read	writer	View workflows, audit, recent copies, AI rule suggester. No replay, no release, no AI settings
None / 404	denied	401 Unauthorized

write is deliberately mapped to writer, not operator: most writers on docs repos have write on the auth repo, so mapping write → operator would give every writer replay and release capability. Operator actions require an explicit admin or maintain grant.

On top of the role, replay is repo-scoped: the user's PAT must also have read access to the source repo of the webhook being replayed. This prevents someone who has operator access to the copier from replaying webhooks for private source repos they can't read. Checked on every replay call and cached per (token, repo) with a 5-minute TTL.

What's new

Backend APIs

• GET /operator/api/workflows — parsed copier config for the workflow browser
• POST /operator/api/replay — replay a webhook delivery (in-flight dedup, merged-PR check, per-repo permission check)
• GET /operator/api/logs?delivery_id=X — per-delivery log viewer via context-tagged ring buffer
• GET /operator/api/me — authenticated user: {login, avatar_url, role}
• GET /operator/api/repo-permission?repos=... — batch permission check for frontend button state
• POST /operator/api/suggest-rule — LLM-generated copier rule from a source→target example, self-verified against the in-process PatternMatcher
• GET /operator/api/llm/status — provider, reachability, active model, installed models, supports_model_mgmt flag
• POST /operator/api/llm/settings — switch active model / base URL at runtime (in-memory, resets on restart)
• POST /operator/api/llm/pull (Ollama only) — NDJSON progress streaming for model pulls
• DELETE /operator/api/llm/model?name= (Ollama only) — remove a pulled model
• PR number + file path filters on GET /operator/api/audit/events
• CommitSHA added to WebhookTraceEntry and CopierFileMeta

AI rule suggester

• Two providers, one interface — LLMClient interface with concrete ollamaClient and anthropicClient implementations. NewLLMClient dispatches on LLM_PROVIDER.
• Anthropic (hosted, default in prod) — calls /v1/messages through the Grove Foundry APIM gateway. Sends both x-api-key (native Anthropic) and api-key (Azure APIM) headers so the same client works with either endpoint. Ping uses a 1-token /v1/messages call (the gateway doesn't expose /v1/models). ListModels falls back to a static list when the endpoint isn't reachable.
• Ollama (local) — full UI-based management: pull models with streaming progress, delete, switch active. Used for dev or any environment with a reachable Ollama VM.
• Self-verifying — every generated rule is run through the in-process PatternMatcher before display; mismatches surface a warning.

Bug fixes

• ObjectID decoding for mongo-driver v2 (ObjectIDAsHexString) — audit queries work again
• Empty-commit PR creation bug (errTreeUnchanged sentinel) — skips cleanly instead of 422'ing on empty branches
• Keyboard-shortcut overlay wouldn't close — .help-bg[hidden] now wins over the base display:flex
• File match tester returned no matches for Java files — JavaScript RegExp doesn't support Python-style (?P<name>); rewrites (?P< → (?< before compilation

Frontend

• 5-tab layout: Overview, Webhooks, Audit, Workflows, System
• Writer/Operator mode toggle (writers see a trimmed tab set)
• Sticky status bar: version, uptime, MongoDB health, last webhook, connection status, GitHub avatar + username
• Dark mode with full CSS variable theming
• Collapsible sections (persisted per-section in localStorage)
• Grouped metrics with delta indicators, sparklines, health accent borders
• Audit event detail drawer with replay button
• File match tester — client-side glob/regex/move/copy matching with computed target paths
• AI rule suggester panel — source + target paths in, generated rule out, with Verified/Not-Verified badge
• AI settings panel — status line labels the provider ("Anthropic connected at…" vs "Ollama connected at…"); pull/delete sections hidden for hosted providers; provider-specific onboarding when disconnected
• PR lookup combining traces + audit into one timeline
• Recent copies card-based feed
• Workflow browser with search and getting-started help
• Per-delivery log viewer in a side drawer (levels color-coded)
• Replay buttons hidden for writers; disabled with tooltip for source repos the user can't access
• CSV export for audit events
• Shareable state URLs (?tab=audit&pr=123&mode=writer)
• Keyboard shortcuts: 1-5 tabs, R refresh, D dark mode, T time format, W writer/operator, ? help

Security

• Token-based auth removed entirely — GitHub PAT is the only supported mechanism
• gosec G107/G704 SSRF findings fixed: GitHub API URL construction validates path components against strict RE2-compatible whitelists (ghUsernameRe, ghRepoNameRe) and escapes them with url.PathEscape
• slack_notifier.go #nosec annotation extended to cover NewRequestWithContext
• Anthropic API key loaded from GCP Secret Manager in prod (same pattern as webhook secret and Mongo URI); never logged

Deploy prep

• .github/workflows/ci.yml — adds operator UI + AI env vars to the Cloud Run deploy, flips AUDIT_ENABLED to true (aligning with the v0.3.0 "enabled by default" CHANGELOG entry)
• env-cloudrun.yaml — parity with the CI deploy for the manual fallback path
• CHANGELOG.md — [Unreleased] populated so scripts/release.sh vX.Y.Z can extract release notes cleanly
• Startup banner — surfaces provider, model, base URL, auth repo at boot
• cmd/test-llm — smoke-test CLI to verify provider connectivity end-to-end after a key rotation or gateway change

One-time deploy prep (before cutting the next tag)

Create the Anthropic gateway key secret on the prod GCP project and grant the Cloud Run runtime service account read access:

# Secret name must match ANTHROPIC_API_KEY_SECRET_NAME in the deploy env
gcloud secrets create anthropic-api-key --data-file=- \
  --project=github-copy-code-examples <<< "$GATEWAY_KEY"

# Find the runtime SA (or check existing bindings on webhook-secret):
RUNTIME_SA=$(gcloud run services describe examples-copier \
  --region=us-central1 --project=github-copy-code-examples \
  --format='value(spec.template.spec.serviceAccountName)')

gcloud secrets add-iam-policy-binding anthropic-api-key \
  --member=serviceAccount:${RUNTIME_SA} \
  --role=roles/secretmanager.secretAccessor \
  --project=github-copy-code-examples

After merge, ./scripts/release.sh v0.4.0 cuts the tag and the Cloud Run deploy picks up the new provider.

Test plan

Local dev

• Export OPERATOR_UI_ENABLED=true without OPERATOR_AUTH_REPO — startup fails with a clear error
• Export OPERATOR_UI_ENABLED=true + OPERATOR_AUTH_REPO=<repo> — server starts, /operator/ loads
• Paste your GitHub PAT — user chip appears with correct role (operator for admin/write, writer for read)
• As a writer, verify replay/release/AI-settings are hidden
• As an operator, Replay a webhook trace for a repo you have access to — reprocesses cleanly
• Attempt replay for a source repo you can't read — button disabled with tooltip; API returns 403
• File match tester — matching rules highlighted with correct target paths
• AI rule suggester (Anthropic via Grove gateway) — verified via cmd/test-llm -env .env.test; Ping, ListModels, and a real suggester prompt all succeed
• AI rule suggester (Ollama, local) — connect, pull a model, switch active, generate a rule

Hosted (Cloud Run) — after merge

• Create anthropic-api-key secret and grant Secret Manager access to the runtime SA (see above)
• ./scripts/release.sh v0.4.0 → verify Cloud Run deploys the new revision
• Open https://<service>/operator/ — all 5 tabs load with a valid PAT
• AI settings shows "Anthropic connected at https://grove-gateway-prod.azure-api.net/…"
• Generate a rule via the AI suggester — verified badge appears
• Merge a PR that matches a workflow — appears in Webhooks, Audit, and Recent copies
• Click Logs on a trace — log entries appear in the drawer

[dacharyc]

Back to you with a handful of auth/scoping-related issues, and some suggestions to add test coverage for the rule suggester, @cbullinger !

[dacharyc]

GET /operator/api/audit/events is wrapped with wrapAPI, so both RoleOperator and RoleWriter can read it. The handler passes the user-supplied filters straight to QueryAuditEvents with no scoping to the caller's GitHub permissions, so any writer (anyone with read/triage/write on OperatorAuthRepo) sees every audit row across every source/target repo the copier has touched — including repos they have no access to on GitHub.

Each row includes source_repo, source_path, target_repo, target_path, commit_sha, pr_number, error_message, and the free-form additional_data map (audit_logger.go:23). Even just the repo+path tuples can leak internal project names, unreleased work, or service topology to a writer who shouldn't see them.

The building block to fix this already exists in the PR: ghCache.CanUserReadRepo is used at line 276 to gate the per-repo replay button. We should probably post-filter results in handleAuditEvents — drop any event whose source_repo (and/or target_repo) the caller can't read. The existing repo-perm cache absorbs most of the GitHub API cost; worst case is one API call per distinct repo per uncached PAT.

[dacharyc]

AdditionalData map[string]any is serialized to the audit-events JSON response verbatim and rendered in the operator UI. Worth auditing every call site that populates this field (and ErrorMessage) to make sure nothing sensitive lands here — raw GitHub API error bodies, tokens accidentally included in error context, internal IDs, etc. Once /operator/api/audit/events is reachable by the writer role, anything in this map is reachable by them too.

[dacharyc]

Minor follow-on to the authorization issue: if/when audit results get scoped to the caller's accessible repos, consider also accepting an explicit source_repo / target_repo query param. Right now path is the only way to narrow by repo (substring match against source_path/target_path), which is awkward and won't match the indexed source_repo field. It looks like there's already a Mongo index on source_repo (audit_logger.go:153) that's currently unused by this query path.

[dacharyc]

Noting here the same authorization gap as /operator/api/audit/events — wrapAPI lets writers read this, and WebhookTraceEntry (webhook_trace_buffer.go:11) includes Repo, BaseBranch, CommitSHA, PRNumber, plus a free-form 500-char Detail. A writer with no GitHub access to repo X still sees X's webhook activity here, including whatever is stuffed into Detail. Recommend the same fix as audit: post-filter by ghCache.CanUserReadRepo against entry.Repo.

It's probably also worth scanning every AppendWebhookTrace(...) call site (webhook_handler_new.go:91 onward) to confirm Detail doesn't carry tokens or other secrets. The 500-char truncation is a length cap, not a redaction.

[dacharyc]

Lower-severity, but flagging for completeness: this is also wrapAPI. The payload (DeliverySnapshot: delivery_id, seen_at, duplicate) doesn't carry repo info, so a writer can only see "we received some webhooks." Probably fine as-is, but worth deciding intentionally rather than by default.

[dacharyc]

wrapAPI again, lookup by delivery_id. A writer can pivot from /observability/webhook-traces (which leaks delivery IDs from repos they don't have access to) into the per-delivery logs. Whether that matters depends on what DeliveryLogs captures. We might want to do a spot check to confirm log lines don't carry secrets or content that wouldn't otherwise reach a writer.

[dacharyc]

Even with the audit/traces endpoints scoped per repo, this endpoint hands a writer the entire workflow config — every source repo, target repo, and path pattern the copier knows about. If "writer" is supposed to be a scoped role, this is the largest exposure of repo topology in the PR; if the design is "writers are trusted to know what the copier does," that's fine but worth documenting next to RoleWriter (operator_auth.go:63) so the boundary is explicit.

[dacharyc]

I was asking Claude questions about the implementation here, and Claude suggested this fix, so take it with a grain of salt - seems plausible but it's not in my domain expertise: Auth cache + PAT validation are an unbounded amplifier for bad input

ghAuthCache.set / setRepoPerm cache errors at the same TTL as successes (operator_auth.go:120, 151), and the size-triggered sweeps (lines 130, 160) only evict already-expired entries. Combined with the fact that any Bearer string is forwarded to GET /user before validation, an attacker can:

1. Hold ~200 bytes/entry in the cache for 5 min per unique random token they send.
2. Force 1–2 outbound GitHub calls per request, draining the server's GitHub QPS budget.

Two cheap fixes:

• Reject Bearer tokens that don't match a known GitHub PAT prefix (ghp_, github_pat_, gho_, ghs_, ghu_) before any cache or network access.
• Cap absolute map size (LRU or simple "drop random entry on overflow") rather than relying on TTL-based eviction.

Not exploitable into role escalation, but a meaningful DoS amplifier for an internet-exposed operator UI.

[dacharyc]

I'm noting the test coverage here on the suggester is missing some load-bearing pieces. Each transform type has one happy-path verification test, but the coverage misses the parts most likely to break in production:

1. renderRuleYAML is completely untested; this is the artifact the user copies into config. Minimum: one golden test per transform type, plus the req.SourceRepo == "" branch.
2. handleSuggestRule is completely untested — no coverage of the 405/503/429/400/502 branches, the Verified=false warning path, or default field application in askLLMForRule. Easy table-driven tests with a fake LLMClient.
3. Move uses strings.HasPrefix without a path-boundary check (operator_suggest_rule.go:229). from=agg/py against source=agg/python/foo.py would pass verification with rel=thon/foo.py. The prompt doesn't force from to end at a / boundary, so this is reachable. We should probably add a boundary check in verifySuggestedRule or pin the current behavior with a test that demonstrates the divergence between verifier and server-side transform (whichever is correct).
4. No "rule applies but target mismatch" test for glob or regex — only "rule doesn't apply at all." This is the most likely real failure mode for an LLM-generated rule (close-but-wrong template).
5. computeGlobRelativePath corner cases (no wildcard, leading wildcard, brackets) aren't exercised — small table test would lock these down.

#1, #2, and #3 I'd consider blocking; the rest are nice-to-have.

[cbullinger]

@dacharyc Thanks for the thorough review!

Addressed items 1–4 and the blocking suggester pieces in commits 739ab56 and f41bc52.

Addressed

• /audit/events, /observability/webhook-traces, /logs, /workflows — new repoFilter (services/operator_repo_filter.go) post-filters all four endpoints by CanUserReadRepo. Operators bypass; writers only see rows whose referenced repos they can read on GitHub. config_change audit rows (no repo) are operator-only. /logs resolves a delivery's repo via a new WebhookTraceBuffer.RepoForDelivery helper and denies writers when the trace has aged out. /workflows returns a hidden_count so the UI can hint without leaking names. RoleWriter doc updated to make the boundary explicit.
• Move-transform path-boundary bug — verifySuggestedRule now requires the prefix to end at a / boundary, so from=agg/py no longer falsely matches agg/python/.... Regression test added.
• Suggester test coverage — golden renderRuleYAML tests per transform type + empty-source-repo branch; handleSuggestRule branch tests for 405/503/429/400/502, the Verified=false warning path, and askLLMForRule default application (name, branch, dest_repo, commit_strategy).

Deferred to follow-up issues (intentionally — would balloon this PR's diff)

• AdditionalData / ErrorMessage hygiene sweep — call-site audit to confirm nothing sensitive lands in those fields, especially now that they're reachable by writers (after this PR's scoping).
• Auth-cache DoS hardening — PAT prefix check before validation/cache + bounded LRU on ghAuthCache. Plausible mitigation, but separate subsystem.
• Suggester nice-to-haves — glob/regex "rule applies but target mismatch" tests, computeGlobRelativePath corner cases.

[dacharyc]

Cool, thanks for fixing up the blockers I called out - let's get this delightful addition merged so we can all enjoy the beautiful new dashboard 😁