proxy: inject default User-Agent so GitHub-style "UA-required" services don't 403 silently

[eanzhao]

Summary

When a NyxID proxy client doesn't send a User-Agent header, the request is forwarded to the downstream service with no UA. For services that require a UA (notably GitHub: https://docs.github.com/en/rest/overview/resources-in-the-rest-api#user-agent-required), this surfaces as a 403 with body Request forbidden by administrative rules. Please make sure your request has a User-Agent header. This puts a hard upstream requirement on every proxy caller and makes the failure mode opaque (the 403 looks like an auth issue).

We just hit this in production at aevatar: every /daily Lark-bot command failed with 403 github_proxy_access_denied even though the user's GitHub OAuth was healthy and the agent api-key carried the correct UserService.id in allowed_service_ids. Manual reproductions via nyxid proxy request api-github rate_limit from the CLI always returned 200, because Rust reqwest auto-sets reqwest/x.y as its UA. The production .NET HttpClient does not. Workaround landed at aevatarAI/aevatar#421 (inject aevatar-agent-builder at the .NET client boundary).

Current behavior

• backend/src/handlers/proxy.rs:1576-1581 and proxy_service.rs:1903-1929: client's User-Agent is forwarded as-is. custom_user_agent on UserService / DownstreamService overrides it when set.
• If the client sends no UA and custom_user_agent is None, no UA reaches the downstream.

That's a fine default for most APIs but breaks for the ones that require UA per their documented contract.

Proposed change

Inject a default User-Agent (e.g. NyxID-Proxy/{version}) at the proxy boundary when all of:

1. The client request has no User-Agent header, AND
2. The resolved UserService.custom_user_agent is None (don't overwrite explicit per-user config), AND
3. The resolved DownstreamService.custom_user_agent (if catalog-backed) is None.

This way every proxied request has some UA, downstream services that require it stop 403'ing for a non-obvious reason, and existing customizations still win. Non-UA-requiring services are unaffected (any non-empty UA is fine).

Open question: should the injected UA include the calling user / api-key id (e.g. NyxID-Proxy/{version} (key=<prefix>))? Probably no — gives upstreams more correlation power than they need, and the existing per-service custom_user_agent covers identification cases.

Why "every client should send UA" is fragile

• .NET HttpClient doesn't send one by default.
• Many languages' default HTTP clients don't (Python urllib, Java HttpURLConnection, etc.).
• The proxy is a better enforcement point than every SDK.

Repro

# Create a key with narrow proxy scope:
TOKEN=$(cat ~/.nyxid/access_token)
KEY_VALUE=$(curl -sS -X POST https://nyx-api.chrono-ai.fun/api/v1/api-keys \
  -H \"Authorization: Bearer $TOKEN\" -H \"Content-Type: application/json\" \
  -d '{\"name\":\"ua-repro\",\"scopes\":\"proxy\",\"allowed_service_ids\":[\"<your api-github UserService.id>\"],\"allow_all_services\":false}' \
  | python3 -c 'import json,sys; print(json.load(sys.stdin)[\"full_key\"])')

# Call without UA — gets 403
curl -sS -i \"https://nyx-api.chrono-ai.fun/api/v1/proxy/s/api-github/rate_limit\" \\
  -H \"Authorization: Bearer $KEY_VALUE\" \\
  -H \"User-Agent:\"     # explicit empty — bash strips it; for an actual repro use a .NET/Python/Java client

Inversely, the same key with reqwest works because reqwest auto-sets a UA.

Relevant code paths

• Proxy header forwarding: backend/src/services/proxy_service.rs:1900-1932
• HTTP-path UA override: backend/src/handlers/proxy.rs:1576-1581
• Existing tests:
  • forward_request_passes_through_user_agent_by_default (proxy_service.rs:2631)
  • forward_request_overrides_user_agent_when_service_has_custom (proxy_service.rs:2680)
• Add a third test for the proposed behavior: forward_request_injects_default_user_agent_when_client_omits_one_and_no_custom_set.

Acceptance

• Client request to /proxy/s/api-github/rate_limit with no User-Agent returns 200 (GitHub processes the request normally) instead of 403.
• Client request that does set User-Agent: foo/1.0 still forwards foo/1.0 (no override).
• UserService.custom_user_agent and DownstreamService.custom_user_agent continue to override both the client UA and the new default.

Related

• fix(nyxid-proxy): inject default User-Agent so GitHub doesn't 403 aevatarAI/aevatar#421 — workaround at the .NET client boundary (lands the UA injection at NyxIdApiClient.ProxyRequestAsync). Catches the immediate aevatar callers but doesn't help future SDKs/clients in other languages.