feat: ship MCP 0.4 with internal API routing

[ash0ts]

Why

MCP 0.4 needs to reach W&B through the in-cluster Service instead of sending backend API traffic through the public load balancer. It also needs an explicit customer runtime contract that does not inherit Gorilla administration settings or Kubernetes API credentials.

What changes

• release operator-wandb 0.44.6 with MCP server 0.4.0
• keep WANDB_BASE_URL public for links and credential lookup
• route backend W&B calls internally:
  • split API: http://<release>-api:8081
  • monolith: http://<release>-app:8080
• require an explicit WANDB_INTERNAL_BASE_URL for custom or external topologies
• isolate MCP from global administrative and Gorilla environment variables through wandb-base.globalOptOut.env
• disable MCP service-account token mounting; MCP receives no Kubernetes API credential
• keep one replica, one worker, HPA/KEDA off, and the existing admission/deadline limits
• keep raw GraphQL, Weave Agent, and ARIA tools disabled by default for Dedicated
• add no database, node pool, Service, HPA, RBAC rule, or other new Kubernetes resource

The generic wandb-base opt-out requires the mechanical dependency releases wandb-base 0.12.7, lumen 0.2.6, and orchestrator 1.4.4.

Dedicated contract

• MCP_HOSTED_MODE=true retains the hard HTTP-service history and trace-query gates
• MCP_WORKLOAD_PROFILE=dedicated
• 17 tools without classic Weave; 22 when classic Weave is available
• writes enabled
• raw GraphQL, Weave Agent, and ARIA tools disabled by default
• one replica and one worker with Dedicated admission and rate limits
• public user-visible links and internal server-side W&B API routing

Operator-supplied W&B and Weave backend URLs are validated before rendering. Custom or ambiguous topologies fail closed rather than silently targeting a nonexistent or unsafe Service.

Safety

• no GORILLA_MCP_* settings or dependency on the deferred Core workload lane
• no Kubernetes API token in the MCP pod and no MCP RBAC policy
• no inherited admin API key or Gorilla server-only environment
• no Minecart, cloud IAM, or deployment authority in the customer workload
• feature settings remain safe, overrideable Helm defaults
• rollback remains a normal Helm rollback or disabling mcp-server.install

Validation

Using the repository-pinned Helm 3.20.1:

• 123/123 Helm unit tests passed across seven suites
• 74/74 focused MCP deployment, health, and validation tests passed
• all operator-wandb snapshots matched
• dependency cascade build and Helm lint passed
• digest rendering is covered explicitly and produces repository@sha256:...
• current-head GitHub CI/security checks must pass before review

Tests cover split API, monolith, custom releases/namespaces, explicit overrides, disabled MCP, every size, exact feature profiles, global-environment isolation, malformed URLs, fail-closed topology errors, disabled service-account token mounting, absence of MCP RBAC, and immutable image rendering.

Remaining release gates

Keep this PR draft and unmerged until:

• the final MCP 0.4.0 customer image is published and this chart pins its verified immutable digest; the public 0.4.0 tag is not published yet, so no historical candidate digest is used here
• the customer registry path can pull that exact digest
• Dedicated Server 0.82.x proves W&B traffic reaches the internal ClusterIP with no corresponding public-ALB API traffic
• 17/22-tool manifests, active and exported history, registry reads, public links, privacy, load, upgrade, and rollback all pass
• every current-head GitHub check is green and normal human approvals are present

No production promotion, customer deployment, Helm merge, or Core workload isolation is performed by this PR.

[ash0ts]

Refreshed validation at 9d5bfd9fb9755485aade71c88a519daae8327131

• merged current main into the draft branch without conflict
• Helm 3.20.1 dependency build passed
• all operator snapshots passed, including every MCP routing/profile/override case
• MCP values linted cleanly
• default and custom release/namespace renders correctly keep WANDB_BASE_URL public and generate WANDB_INTERNAL_BASE_URL=http://<release>-api:8081
• explicit internal URL override, image 0.4.0, one worker, admission enabled, and raw GraphQL disabled all rendered correctly
• Socket, lint, release, snapshot, and every non-OLAP chart job passed

The three olap-features-enabled jobs fail on Kubernetes 1.34/1.35/1.36 in the existing Weave/OLAP migration/readiness path. That fixture does not enable MCP, this PR changes no OLAP/Weave files, and the logs show the timeout in weave-trace-migrate / Weave readiness. This is unrelated to the MCP chart changes and has not been skipped or weakened.

No chart was installed into a real cluster.

[ash0ts]

CI diagnosis for pushed SHA 3832c6b8ab61fa077a5a18e92f3126412540c956:

• snapshot, lint, release, Socket Security, and every completed non-OLAP chart job passed
• the three failures are only olap-features-enabled on Kubernetes 1.34/1.35/1.36
• that fixture renders no MCP Deployment, Service, or MCP environment configuration
• each job spent the full ten-minute install window waiting while the existing parquet/Weave containers restarted during startup; logs show parquet BackOff/restart activity and later healthy Weave /traces/health responses
• no MCP-owned template is present in the failing fixture, so this is the existing OLAP readiness failure rather than a workload-lane regression

No unrelated retry/readiness workaround is included in this PR. The draft remains blocked on Core #49143 and the MCP 0.4.0 image as described above.

[ash0ts]

Exact-SHA validation checkpoint

Validated head: d2f80268cabbfc32973c5e548e2027006b6d6b65, rebased on main f1e91050356fe157d54ebea60db04b7353923032.

• dependency build, helmfmt, template-unit/maintainability tests, lint, schema/YAML checks, every repository snapshot, and focused release/namespace/internal-URL/profile/capacity renders passed locally
• GitHub lint, release, snapshot, CodeRabbit, and Socket Security checks passed
• Kubernetes matrix 30557048152 passed 40/43 jobs
• only the three olap-features-enabled installs failed, across Kubernetes 1.34.8, 1.35.5, and 1.36.1: ClickHouse reports Database artifacts does not exist (UNKNOWN_DATABASE), weave-trace-migrate backs off, and the install reaches its 600-second deadline

That fixture does not enable MCP, and recent unrelated chart PRs fail the same three jobs. This is still a hard merge blocker: no bypass was used, and the PR remains draft pending an upstream fix or clean rerun. Exact published MCP 0.4.0 image-digest validation is also still required.

[ash0ts]

Release-gate correction: this PR no longer depends on Core #49143, MCP #125, or Helm #666 and contains no GORILLA_MCP_* settings. The remaining blockers are (1) the three red olap-features-enabled matrix jobs, (2) validation of the exact published MCP 0.4.0 image digest in Dedicated staging, and (3) delivery-tooling approval. Keep this PR draft and unmerged until all three pass.

[ash0ts]

Topology gate fixed at ba0034025fbd7ccb6ab9e9806173762deea1f118:

• split API now renders WANDB_INTERNAL_BASE_URL=http://<release>-api:8081
• monolith now renders WANDB_INTERNAL_BASE_URL=http://<release>-app:8080
• when the selected internal Service is disabled, an explicit non-empty override is required
• all MCP-only validation is skipped when mcp-server.install=false
• invalid 99/100 admission test values were replaced with supported 12/24 values
• new CI render tests cover both topologies, external overrides, missing Services, disabled MCP, custom release/namespace, all size profiles, runtime guardrails, and absence of rendered GORILLA_MCP_*

Local results: Helm 3.20.1 dependency build, strict Helm lint, chart-testing schema/YAML lint, 8 focused MCP render tests, 16 template-tool tests, maintainability/format checks, and all operator snapshots passed. PR remains draft and unmerged. The exact-head GitHub matrix is now running; upstream OLAP fix candidate #669 has all three OLAP install jobs green but still has its own snapshot failure.

[ash0ts]

Staff-review follow-up at 3cea48cb7e8b70d1c2fe8af2d5b4462eafb0e01c closes the custom-Service routing gap.

The chart can safely auto-resolve only the default child-chart endpoints (<release>-api:8081 and <release>-app:8080). The MCP env helper runs in MCP subchart scope, so attempting to duplicate the API/app child charts’ alias, fullname, and tpl behavior would be brittle. The chart now fails closed and requires an explicit non-empty mcp-server.env.WANDB_INTERNAL_BASE_URL whenever API/app service.name, nameOverride, fullnameOverride, or the backend Service port is customized. This validation remains gated by mcp-server.install=true.

Validated with Helm 3.20.1: 9 focused routing tests (including every custom-name/port case with and without an explicit URL), all operator snapshots, Helm lint, chart-testing schema/YAML lint for wandb-base and operator-wandb, 16 template-tool tests, helmfmt check, template-maintainability check, and git diff --check. No chart was installed or merged.

[ash0ts]

Exact-head CI checkpoint for 3cea48cb7e8b70d1c2fe8af2d5b4462eafb0e01c:

• 55 checks are green, including lint, release packaging, Socket, the other chart snapshots, and 39/42 operator Kubernetes matrix jobs.
• The only operator matrix failures are the same three unrelated olap-features-enabled jobs on Kubernetes 1.34.8, 1.35.5, and 1.36.1.
• The operator snapshot job initially failed before executing tests because Helm cascade received an empty transient prometheus-index.yaml. Its focused rerun is queued as job 91201253332. Local Helm 3.20.1 snapshots and the focused MCP suite passed.

PR remains draft, unmerged, and correctly blocked on the OLAP release gate plus Dedicated staging validation.

[ash0ts]

Exact-head CI update for 3cea48cb7e8b70d1c2fe8af2d5b4462eafb0e01c: 57 checks are green, including the retried snapshot job. The only failures are the three olap-features-enabled jobs. Upstream PR #669 demonstrates the fix and passed that matrix on Kubernetes 1.34.8, 1.35.5, and 1.36.1. This PR should rebase after #669 merges; it remains draft pending that rebase and exact-image Dedicated staging validation.

[ash0ts]

v0.4 release evidence for the eventual Dedicated gate: public SHA 6a465a34b0d3c63c881a244004c4b97088a45b5d, hosted staging SHA dc4682931835f8a3d4633cdb84bb336f824b1eca, Cloud Run staging digest sha256:4c04361578d60787dce07822bcfbca46f34ec815004af820acc43b3ee499f0a9. The public/hosted candidate passed 28/28 read acceptance, exact writes/cleanup, 23/23 telemetry/privacy, same-key burst, normal load, and a paired W&B API canary. This Cloud Run digest is evidence for the code candidate, not a substitute for the published customer image digest or Dedicated ClusterIP/no-public-ALB trace. #657 remains draft pending #669 merge/rebase and that real Dedicated validation.

[ash0ts]

Server Release handoff: one v0.4 gate

Head 51becb3735ea7a9fd9368f19a02fea559ac466ce adds a single MCP v0.4 release gate tied to the exact accepted artifact:

• public MCP source: 6a465a34b0d3c63c881a244004c4b97088a45b5d
• hosted wrapper source: dc4682931835f8a3d4633cdb84bb336f824b1eca
• staging-tested image digest: sha256:4c04361578d60787dce07822bcfbca46f34ec815004af820acc43b3ee499f0a9
• public destination: us-docker.pkg.dev/wandb-production/public/wandb/mcp-server:0.4.0

The Server Release team has one remaining action: promote that exact digest to the public destination/tag without rebuilding it. The gate then verifies tag-to-digest equality, package version and embedded public source SHA, exact Helm tag/digest rendering, least-privilege/runtime settings, local container health, a Kind rollout, the running pod image ID, and /mcp/health. It uploads the rendered manifest and runtime evidence.

The public 0.4.0 tag is not present yet, so this check is expected to fail closed until that promotion happens.

Also fixed the chart's Helm test hook from the wrong /health path to the canonical /mcp/health path.

The unrelated OLAP installs are no longer blindly used as an MCP release blocker. A new fail-closed preflight renders base and head for Kubernetes 1.34/1.35/1.36 and excludes OLAP only when all normalized manifests are byte-identical. Any semantic difference or uncertain comparison retains the full OLAP matrix. For this head, all three OLAP renders are identical. No code from #669 was copied into this PR.

[ash0ts]

Scope reduced for review

Rebased head: 01ceaaf5f14dcf3fcb48d7b090da3983f77ae4a1 on current main e49742bf.

The net PR is now 26 files / +900 / -159, down from 39 files / +24,352 / -177.

Removed from this Helm feature PR:

• the version-specific MCP release-gate workflow and hard-coded release-candidate metadata
• the repository-wide OLAP matrix-selection change
• four redundant full-manifest MCP snapshots (over 22K generated lines)
• unrelated workflow hardening

Retained because it directly supports the customer deployment:

• operator-wandb 0.43.11 and MCP image 0.4.0
• public WANDB_BASE_URL plus topology-aware internal routing:
  • split API: http://<release>-api:8081
  • monolith: http://<release>-app:8080
  • explicit override for custom topology
• bounded admission/timeouts, one MCP replica/worker, and explicit tool-surface flags
• canonical /mcp/health Helm test
• least-privilege MCP environment isolation

The wandb-base/Lumen/Orchestrator version updates are mechanical consequences of the least-privilege globalOptOut.env addition. Keeping that behavior prevents global admin/Gorilla variables from reaching the MCP pod; dropping it would make the diff smaller by weakening the deployment boundary.

Validation on the reduced tree:

• focused routing/runtime/least-privilege renders: 6/6 passed
• Helm 3.20.1 strict lint: wandb-base, Lumen, Orchestrator, operator-wandb passed
• all snapshots for those four charts passed
• dependency lock regeneration passed
• template format/maintainability checks and 16 repository script tests passed
• git diff --check passed

Release SHA/digest evidence remains in the release-control process/PR discussion rather than permanent v0.4-specific code in this chart repository.