Add compatibility enforcement for public WebSocket, SSE, and other non-OpenAPI surfaces

[enyst]

Problem

Our REST/OpenAPI compatibility enforcement only covers the public API surface that appears in OpenAPI.

That leaves a gap for public compatibility-sensitive surfaces such as:

• WebSocket endpoints
• SSE/streaming endpoints
• other externally consumed message/event shapes not represented in OpenAPI

The current agent-server policy explicitly notes that WebSocket/SSE endpoints are not covered by the OpenAPI-based policy.

What we may want

Define and enforce a compatibility policy for these non-OpenAPI public surfaces, for example:

• snapshot/schema tests for WS/SSE message payloads
• explicit versioned event/message schemas
• deprecation metadata for externally visible streaming/event contracts
• targeted compatibility fixtures for clients

Why

If these surfaces are public, they deserve the same kind of compatibility discipline as the REST API. Right now they are effectively outside our automated enforcement model.

[enyst]

@OpenHands check all AGENTS.md and griffe and oasdiff API compatibility workflows. Think about this issue. You can see those two workflows use each a particular library suited for the format/checks. WDYT, investigate, research the internet or github and come up with proposals on this surface. Post directly as a comment in response to this issue.

[openhands-ai]

I'm on it! enyst can track my progress at all-hands.dev

[enyst]

@enyst I checked the AGENTS files and the existing compatibility workflows.

My read is:

• griffe works for the SDK because there is an explicit Python-symbol public surface.
• oasdiff works for agent-server REST because there is an explicit OpenAPI contract.
• WS/SSE/non-OpenAPI surfaces are missing that equivalent machine-readable contract today, so the main gap is not just “another diff step”, it is “what canonical contract format are we diffing?”.

A few concrete findings from the current repo:

• The agent-server AGENTS file explicitly excludes WebSocket/SSE from the current REST/OpenAPI policy.
• The current public non-OpenAPI surface in this repo seems to be mainly the two WS endpoints under openhands.agent_server.sockets:
  • /sockets/events/{conversation_id}
    • connect/query/header surface: session_api_key, resend_mode, after_timestamp, deprecated resend_all
    • client -> server payload: Message
    • server -> client payload: openhands.sdk.Event
  • /sockets/bash-events
    • connect/query/header surface: session_api_key, resend_mode, deprecated resend_all
    • client -> server payload: ExecuteBashRequest
    • server -> client payload: BashEventBase
• I did not find a public SSE endpoint in openhands-agent-server today.
• We already have useful WS tests (tests/agent_server/test_agent_server_wsproto.py, test_event_router_websocket.py), but they are behavior/regression tests, not release-to-release compatibility enforcement.

Proposal A (my recommendation): AsyncAPI for transport/contracts + fixtures for runtime validation

This feels like the closest analogue to the existing griffe / oasdiff setup:

1. Add a small declarative registry for public non-OpenAPI contracts (initially just the two WS endpoints).
2. Generate an AsyncAPI document from that registry / from the actual Pydantic models, rather than hand-maintaining a spec.
3. In CI, generate the current AsyncAPI doc and the baseline doc from the previous release tag (same pattern as the REST breakage job), then diff them with asyncapi/diff.
4. Apply the same style of compatibility policy we already use elsewhere:
   • channel/path removals require a deprecation runway
   • removed query params / message variants are breaking unless versioned/deprecated first
   • transport-visible behavior like close codes should be covered by targeted contract fixtures
5. Add a few real end-to-end WS contract fixtures on top of the spec diff.

Why I like this path:

• AsyncAPI is the closest thing to “OpenAPI for evented / WS-style surfaces”.
• It models channels, operations, messages, and protocol bindings (including WS query params), which pure schema snapshots do not.
• It preserves the repo’s current pattern of “pick the checker that matches the contract format”:
  • Python API -> griffe
  • REST/OpenAPI -> oasdiff
  • evented/WS contract -> AsyncAPI + AsyncAPI-native diff

Proposal B (smaller first step): JSON Schema compatibility on the payloads

If we want something more Python-native / lower ceremony first, I think a good minimal version is:

1. Define the public WS surfaces explicitly in code.
2. Generate JSON Schema from the actual Pydantic message/event models.
3. Check compatibility directionally:
   • client -> server messages: the new server must still accept everything the old server accepted (old_schema <: new_schema)
   • server -> client messages: everything the new server emits must still satisfy the old client contract (new_schema <: old_schema)
4. Use targeted runtime fixtures for paths/params/close codes.

A library like jsonsubschema is interesting here because it is explicitly about JSON Schema subschema / backward-compatibility reasoning, which is closer to what we want than raw textual snapshots.

The tradeoff is that this still leaves transport-level changes (channel/path changes, bindings, request/reply shape, etc.) partly outside the schema diff, so I would view this as a phase-1 bootstrap, not the final design.

What I would not do

• I would not stop at raw snapshot tests / golden JSON payload dumps. They are useful as fixtures, but by themselves they are brittle and do not give us semantic “breaking vs non-breaking” reasoning.
• I would not wait for OpenAPI to solve WebSockets. For HTTP streaming / SSE specifically, OpenAPI 3.2’s new streaming support (itemSchema / related work) is relevant, but it does not solve the WebSocket gap and tooling adoption will lag anyway.

Pragmatic rollout I’d suggest

Phase 1

• Scope only the two existing WS endpoints.
• Define them explicitly as the public non-OpenAPI surface.
• Add generated contract output + CI breakage check.
• Add a few contract fixtures for:
  • auth/connect params
  • deprecated resend_all
  • resend_mode / after_timestamp
  • representative Event and BashEventBase variants
  • public close codes (4001, 4004)

Phase 2

• If/when public SSE / NDJSON / multipart stream endpoints land, fold them into the same “non-OpenAPI public contracts” system.
• For HTTP streaming specifically, reevaluate whether OpenAPI 3.2 tooling is mature enough to move those under the OpenAPI workflow instead of the AsyncAPI/custom path.

Net/net

If we want this to feel consistent with the existing compatibility workflows, my vote is:

• AsyncAPI as the contract format for WS/SSE/evented public surfaces
• asyncapi/diff (or equivalent) as the transport-level breakage checker
• runtime contract fixtures for wire behavior not fully captured by the spec
• optionally JSON Schema subschema checks for payload-level directional compatibility if AsyncAPI diff alone is not strict enough

That gives us the same overall shape we already have today: one explicit public surface, one format-native checker, and a small amount of targeted runtime coverage.

References / research:

• AsyncAPI WebSocket tutorial: https://www.asyncapi.com/docs/tutorials/websocket
• AsyncAPI diff: https://github.com/asyncapi/diff
• Specmatic AsyncAPI/WebSocket contract testing: https://docs.specmatic.io/supported_protocols/asyncapi/web-sockets.html
• JSON subschema compatibility checker: https://github.com/IBM/jsonsubschema
• OpenAPI streaming / SSE discussion; closed as added in 3.2: An SSE Proposal OAI/OpenAPI-Specification#4171

[openhands-ai]

Final summary:

• Completed the requested investigation of:

  • repository AGENTS.md guidance
  • the existing griffe-based Python API compatibility workflow
  • the existing oasdiff-based REST/OpenAPI compatibility workflow
  • the current non-OpenAPI public surface in openhands-agent-server
  • external tooling/options for WS/SSE/event compatibility

• Posted the requested response directly on issue Add compatibility enforcement for public WebSocket, SSE, and other non-OpenAPI surfaces #2441:

  • Add compatibility enforcement for public WebSocket, SSE, and other non-OpenAPI surfaces #2441 (comment)

• Main recommendation I gave on the issue:

  • use AsyncAPI as the canonical contract format for public WebSocket/SSE/evented surfaces
  • diff release-to-release changes with asyncapi/diff
  • supplement that with a small set of runtime contract fixtures for behavior not fully captured by the spec
  • possible smaller first step: directional JSON Schema compatibility checks using generated schemas, potentially with jsonsubschema

Checklist:

• Reviewed all relevant AGENTS guidance
• Reviewed griffe/oasdiff compatibility workflows and their fit
• Investigated current WS/non-OpenAPI surfaces in this repo
• Researched ecosystem options/tools
• Posted a proposal comment directly on the GitHub issue

Conciseness:

• No repository files were changed.
• No extraneous code/config/test changes were made.
• Since this was an investigation/comment task only, there was nothing to branch, commit, push, or open as a PR.