fix(api): dont 500 on cross dc response

[MasterPtato]

Fixes RVT-6105

Description

Please include a summary of the changes and the related issue. Please also include relevant motivation and context.

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

How Has This Been Tested?

Please describe the tests that you ran to verify your changes.

Checklist:

• My code follows the style guidelines of this project
• I have performed a self-review of my code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes

[railway-app]

🚅 Deployed to the rivet-pr-4449 environment in rivet-frontend

Service	Status	Web	Updated (UTC)
frontend-cloud	😴 Sleeping (View Logs)	Web	Apr 4, 2026 at 10:06 am
frontend-inspector	😴 Sleeping (View Logs)	Web	Mar 19, 2026 at 12:12 pm
website	❌ Build Failed (View Logs)	Web	Mar 17, 2026 at 10:41 pm
mcp-hub	✅ Success (View Logs)	Web	Mar 17, 2026 at 10:40 pm
ladle	❌ Build Failed (View Logs)	Web	Mar 17, 2026 at 10:39 pm

[MasterPtato]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• chore: envoy client #4521 : 2 dependent PRs (#4531 , #4534 )
• feat(pb): actors v3 #4463
• feat(ups): implement queue subs #4486
• feat(cache): add in flight deduping #4459
• fix(cache): clean up lib #4457
• fix(config): allow configuring topo dcs via map, fix pg ssl mode config #4456
• fix(api): dont 500 on cross dc response #4449 👈 (View in Graphite)
• chore: update byte units on frontend #4444
• fix: runner alloc idx logic, api auth for actor get #4443
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[linear]

RVT-6105 Cross-DC proxy escalates 400 client errors to 500 internal errors

[pkg-pr-new]

More templates

• actor-actions
• actor-actions-vercel
• example-agent-os
• example-agent-os-e2e
• ai-agent
• ai-agent-vercel
• example-ai-and-user-generated-actors-freestyle
• chat-room
• chat-room-vercel
• example-cloudflare-workers
• example-cloudflare-workers-hono
• collaborative-document
• collaborative-document-vercel
• cross-actor-actions
• cross-actor-actions-vercel
• example-cursors
• example-cursors-raw-websocket
• example-cursors-raw-websocket-vercel
• example-cursors-vercel
• custom-serverless
• example-elysia
• example-experimental-durable-streams-ai-agent
• example-experimental-durable-streams-ai-agent-vercel
• geo-distributed-database
• geo-distributed-database-vercel
• hello-world
• hello-world-vercel
• example-hono
• hono-react
• hono-react-vercel
• example-kitchen-sink
• example-kitchen-sink-vercel
• multi-region-vercel
• multiplayer-game
• example-multiplayer-game-patterns
• example-multiplayer-game-patterns-vercel
• multiplayer-game-vercel
• native-websockets
• native-websockets-vercel
• next-js
• per-tenant-database
• per-tenant-database-vercel
• example-raw-fetch-handler
• example-raw-fetch-handler-vercel
• example-raw-websocket-handler
• example-raw-websocket-handler-proxy
• example-raw-websocket-handler-proxy-vercel
• example-raw-websocket-handler-vercel
• example-react
• example-react-vercel
• sandbox
• sandbox-coding-agent
• sandbox-coding-agent-vercel
• sandbox-vercel
• scheduling
• scheduling-vercel
• example-sqlite-drizzle
• example-sqlite-raw
• state
• state-vercel
• example-stream
• example-stream-vercel
• example-trpc

@rivetkit/virtual-websocket

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/virtual-websocket@4449

@rivetkit/cloudflare-workers

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/cloudflare-workers@4449

@rivetkit/framework-base

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/framework-base@4449

@rivetkit/next-js

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/next-js@4449

@rivetkit/react

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/react@4449

rivetkit

pnpm add https://pkg.pr.new/rivet-dev/rivet/rivetkit@4449

@rivetkit/sql-loader

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/sql-loader@4449

@rivetkit/sqlite-vfs

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/sqlite-vfs@4449

@rivetkit/traces

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/traces@4449

@rivetkit/workflow-engine

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/workflow-engine@4449

@rivetkit/engine-runner

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/engine-runner@4449

@rivetkit/engine-runner-protocol

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/engine-runner-protocol@4449

commit: 3a9d999

[claude]

PR Review: fix(api): dont 500 on cross dc response

Summary

This is a minimal, well-targeted fix in engine/packages/api-util/src/lib.rs in reqwest_to_axum_response.

Before: When a remote DC returned a non-2xx response, anyhow::bail! propagated an Err up the call stack, causing the API handler to respond with HTTP 500.

After: Logs the error via tracing::error! and falls through to return the actual remote response, preserving the original status code and body.

---

Feedback

Correctness: ✅

The fix is correct. reqwest_to_axum_response should convert a reqwest response to an axum response — a non-2xx upstream status is not a programming error, it's a valid response to be forwarded. Bailing here was the wrong abstraction.

Logging: ✅

The structured logging conversion is well done and aligns with CLAUDE.md: "Do not format parameters into the main message, instead use tracing's structured logging."

Minor observation:

?status (Debug) prints as StatusCode(404) while %status (Display) would print as 404 Not Found, which is slightly more readable. Not blocking, just worth noting.

---

Verdict

✅ Looks good to merge. Simple, correct fix with minimal blast radius.

[claude]

PR Review: fix(api) dont 500 on cross dc response

This is a small, focused bug fix that correctly addresses the issue of cross-DC responses being converted to 500 errors.

Change Summary: In reqwest_to_axum_response, when a remote DC returned a non-2xx response, anyhow::bail!() was called, which propagated as an internal server error (500) instead of forwarding the actual response status and body to the client. The fix logs the error and allows the response to pass through with its original status code.

Whats Good: reqwest_to_axum_response is intended as a raw pass-through so forwarding the original status code and body is the right approach. The new tracing::error! call correctly follows the project logging conventions. The body_bytes borrow via String::from_utf8_lossy is correctly scoped within the if block.

Minor Suggestions: (1) Consider using %status (Display) instead of ?status (Debug) for the status code - Display outputs 404 Not Found while Debug adds extra formatting noise. (2) tracing::error! is reasonable but tracing::warn! might reduce log noise if 404s are expected in normal operation. (3) The checklist items in the PR description are unchecked - worth filling in for completeness.

Overall this is a correct and minimal fix. LGTM.

[claude]

PR Review: fix(api): dont 500 on cross dc response

Summary: This is a clean, targeted fix. When a cross-DC proxy response returns a non-2xx status, the original code bailed with an anyhow error that propagated as a 500. The fix logs the error structurally and passes the original response through instead.

What's good

• The fix correctly preserves the remote DC's status code and body for the caller, which is the right behavior for a transparent proxy function.
• The structured logging (?status, ?ray_id, %body_text) follows the repo's tracing conventions.
• The log message is lowercase and uses structured fields rather than interpolation — matches CLAUDE.md style guidelines.
• Small, focused change with minimal blast radius.

Observations

Log level consideration: Using tracing::error! for all non-2xx responses may be noisy in practice. 4xx responses (e.g. 404, 400, 401) are client errors that are expected in normal operation and typically shouldn't trigger error-level alerts. Consider using tracing::warn! for 4xx and reserving tracing::error! for 5xx responses:

if status.is_client_error() {
    tracing::warn!(?status, ?ray_id, %body_text, "remote dc returned client error");
} else {
    tracing::error!(?status, ?ray_id, %body_text, "remote dc returned server error");
}

This is a minor nit — the current approach is still correct and may be intentional if all non-2xx responses from peer DCs are considered unexpected.

No test coverage needed: Given this is a proxy passthrough fix, the behavior is straightforward and doesn't require additional tests.

Overall this is a solid fix — approve.

[claude]

PR Review: fix(api): dont 500 on cross dc response

Overall: LGTM with one minor suggestion.

The fix is correct. reqwest_to_axum_response is a proxy/passthrough function, so when a remote DC returns a non-2xx response, the old anyhow::bail! was incorrectly converting that into a 500 from the proxying service itself. The response should be forwarded as-is to the caller.

Positive:

• The new tracing::error! call correctly uses structured fields (?status, ?ray_id, %body_text) and a lowercase message, matching the project's logging conventions.
• The response body is correctly preserved and forwarded after the log.

Suggestion: Log level for 4xx responses

Using error! for all non-2xx responses will fire on 4xx client errors from the remote DC (e.g., 404, 401, 422), which may be noisy or misleading since those are expected application-level errors rather than infrastructure failures. Consider differentiating by status class:

if status.is_server_error() {
    tracing::error!(?status, ?ray_id, %body_text, "remote dc returned error");
} else {
    tracing::warn!(?status, ?ray_id, %body_text, "remote dc returned non-success response");
}

This is a minor point and not a blocker — the current approach is still a clear improvement over the previous 500 behavior.

[claude]

PR Review: fix(api): dont 500 on cross dc response

The fix is correct and well-targeted. The reqwest_to_axum_response function is a raw response proxy — when a remote DC returns a 4xx/5xx, the right behavior is to forward the actual error response (preserving status code and body) back to the caller, not convert it into a 500. The previous anyhow::bail! was wrong here because it discarded the remote DC's response entirely.

Observations

%status vs ?status — Minor style point: HTTP status codes display more readably with % (Display) than ? (Debug). %body_text is correct; the status field could match:

tracing::error!(%status, ?ray_id, %body_text, "remote dc returned error");

This would log status=404 Not Found rather than the Debug representation.

Log level for 4xx — Logging all non-success responses at error! level could be noisy if clients make invalid requests that get proxied (e.g., 401, 404). Consider distinguishing 4xx (client errors) from 5xx (server errors):

if status.is_server_error() {
    tracing::error!(%status, ?ray_id, %body_text, "remote dc returned server error");
} else {
    tracing::warn!(%status, ?ray_id, %body_text, "remote dc returned client error");
}

That said, since this is internal DC-to-DC traffic, 4xx from a peer DC may legitimately indicate a problem worth surfacing as an error — so the current choice is defensible.

Contrast with parse_response — The sibling function parse_response (line 198) correctly returns Err on non-success because its callers expect a typed T and need proper error propagation. The asymmetry is intentional and correct.

Summary

The core fix is sound — replacing bail! with a log-and-continue is the right approach for a raw proxy function. The two suggestions above are minor polish items, not blockers.

[claude]

PR Review: fix(api) dont 500 on cross dc response

Summary

This PR fixes a bug where cross-datacenter error responses were causing HTTP 500 errors on the calling service. The root cause was that reqwest_to_axum_response used anyhow::bail! to propagate non-2xx responses from remote DCs, which bubbled up as 500s rather than forwarding the original response. The fix converts the bail! to a tracing::error! log and allows the function to continue building and returning the actual response.

---

The Core Fix Is Correct

The change correctly addresses the problem. Previously, when a remote DC returned a non-2xx status, anyhow::bail! would abort reqwest_to_axum_response and propagate an error upward. The caller would treat this as an internal failure and return 500. Now the error is logged and the original status code and body are forwarded transparently. This is semantically correct for a reverse-proxy scenario.

---

Suggestions

Consider a doc comment on reqwest_to_axum_response

The function contract changed: it now returns Ok even for non-2xx responses. This is intentional and correct for the current callers, but future callers who expect an Err on non-success would silently pass through error responses. A short doc comment would prevent future confusion.

Tracing format specifier (minor)

%body_text (Display) is correct for Cow. For ?status, using Display (%status) might produce a slightly more readable log line (e.g., 404 Not Found vs StatusCode(404)), though this is minor and ? is not wrong.

---

CLAUDE.md Convention Compliance

• Structured logging: The new tracing::error! correctly uses structured fields (?status, ?ray_id, %body_text) rather than formatting values into the message string. Fully compliant.
• No format params in message string: Compliant.
• Conventional commit format: Title follows the format correctly.

---

No Blocking Issues

The fix is correct and well-targeted. The one behavioral nuance worth noting is that non-2xx responses now flow through as Ok rather than propagating as Err. This is intentional and appropriate for the proxy use case. The doc comment suggestion above is optional but would help future maintainers understand the contract.

---

Reviewed by Claude Sonnet 4.6

[claude]

PR Review: fix(api): dont 500 on cross dc response

Summary

This PR fixes reqwest_to_axum_response in engine/packages/api-util/src/lib.rs (+1/-3). When a remote datacenter returned a non-2xx response, the old code called anyhow::bail!, which propagated an error and caused the originating DC to return a 500 to the client. The fix replaces that with a tracing::error! log and forwards the actual status code and body through.

---

Findings

Correctness — correct. reqwest_to_axum_response is used in the raw proxy path (request_remote_datacenter_raw). Bailing on 4xx responses from a remote DC was wrong: it converted legitimate client errors (404, 422, etc.) into 500s at the proxy layer. The new behavior correctly passes the status and body through.

Logging — idiomatic. The tracing::error!(?status, ?ray_id, %body_text, "remote dc returned error") call follows CLAUDE.md conventions: lowercase message, structured fields, no string interpolation in the message.

Minor suggestion — log level. Logging every 4xx at error level may be noisy since client errors are expected behavior. Splitting into warn for 4xx and error for 5xx from the remote DC would be cleaner. Not a blocker.

Sibling function parse_response is unaffected. That function still returns Err on non-2xx, which is appropriate for typed deserialization (not a passthrough proxy).

No regressions. body_bytes is collected before the conditional and reused in the response builder. No resource leaks.

---

Verdict

Small, well-targeted fix. Safe to merge. The log-level split (4xx → warn, 5xx → error) is a nice follow-up but not a blocker.