perf(serve): reduce worker/router event-loop lag

[mikasenghaas]

Summary

Reduce env_worker / env_router event-loop lag. Pairs with the prime-rl PR (perf/r3-rewrite); verifiers changes are on the request-handling side (env_router / env_worker / response unpack).

Coupled with PrimeIntellect-ai/prime-rl#2609 (perf/r3-rewrite). The prime-rl PR fixes the orch-side counterpart of the same lag and bumps the verifiers submodule to this branch's head. Land or revert together — without the prime-rl side the orch loop saturates first under 256-rollout concurrency and these env-side gains don't surface.

What's in here

• parse_response_tokens off the loop (46588cf7, re-added in 68b789ca after a brief revert) — the body now runs in a worker thread (asyncio.to_thread). Was running on the env_worker loop, blocking recv handlers during heavy parse work. One brittle test (test_update_and_reward_children_can_share_borrowed_live_tools) was dropped — it asserted a specific scheduling order against an order-sensitive mock client; the borrowed-tools-sharing contract is still exercised by other tests in the same file.
• state_to_output off the loop (11befd7d) — same idea, env-side. The message serialization to vf.RolloutOutput is CPU heavy enough that running it on the loop noticeably starved other recv tasks. Applied in both Environment.run_rollout and Environment.run_group (group states processed concurrently via asyncio.gather).
• uvloop in env_worker + env_server (4c89d700) — drop-in scheduler-overhead win. Declared as a direct verifiers dep in 31b3830b, with a non-Windows / non-PyPy platform marker in 5d6585c2 (uvloop has no wheels for those).
• Scale thread pool + offload request unpack (445f3797):
  • Bumped the default to_thread executor on env_worker / env_server to 512 — default 32 was bottlenecking 256 concurrent rollouts.
  • msgpack.unpackb + Pydantic model_validate for incoming requests moved to to_thread.
• Cleanup commits: 327fad0e drops intervention-ID prefixes from comments and nests the parse_response_tokens sync body as a closure; f7d7acb0 strips the explanatory perf paragraphs (commit messages carry the why); 34d1a63e ruff fix + format.

What's NOT in here (dropped from earlier iterations)

• env_router on_response asyncio.gather batching (was part of 445f3797, reverted in ed5d78fb) — drained all ready response frames first, then dispatched concurrently. Real wallclock cost for unclear value once the per-rollout offloads landed.

How to validate

Pair with the prime-rl PR and watch the env_worker Lag: line for max=. With these changes the typical max env-worker lag should stay under a few hundred ms even under 256 concurrent rollouts; before this stack we were seeing 6 s+ peaks.

Note

Reduce event-loop lag in serve workers by offloading CPU-bound work to threads

• Offloads state_to_output calls in Environment.run_rollout and Environment.run_group to threads via asyncio.to_thread, with group states processed concurrently using asyncio.gather.
• Offloads msgpack unpacking and Pydantic model_validate in the EnvWorker request handler, and token parsing in parse_response_tokens, to the threadpool.
• Scales the default threadpool executor to concurrency=512 in both EnvWorker and EnvServer, and installs uvloop when available.

Changes since #1453 opened

• Added pytest test validating parallel child rollouts sharing borrowed tools [73e4681]
• Added update and reward functions spawning child rollouts with borrowed tools [73e4681]
• Implemented deterministic test double model client routing responses by prompt content [73e4681]

📊 Macroscope summarized 445f379. 5 files reviewed, 1 issue evaluated, 0 issues filtered, 1 comment posted

🗂️ Filtered Issues

---

Note

Medium Risk
Touches the env server/worker request path and rollout output/token parsing, introducing more asyncio.to_thread usage and larger thread pools; risk is mainly around concurrency behavior, CPU/memory overhead, and subtle ordering/test assumptions.

Overview
Reduces env server/worker event-loop blocking by moving CPU-heavy work off the loop: state_to_output in Environment.run_rollout/run_group, token parsing in parse_response_tokens, and request deserialization (msgpack.unpackb + Pydantic model_validate) in EnvWorker now run via asyncio.to_thread.

Improves runtime scheduling by optionally installing uvloop in EnvServer.run_server/EnvWorker.run_worker and scaling the default executor to handle high to_thread concurrency (set to 512). Updates a lifecycle test to be robust to parallel asyncio.gather interleaving by using a routed mock client and removing order-dependent assertions.

^{Reviewed by Cursor Bugbot for commit 73e4681. Bugbot is set up for automated code reviews on this repo. Configure here.}

[macroscopeapp]

Approvability

Verdict: Approved

Performance optimization that offloads CPU-bound parsing to thread pools and uses uvloop for faster async event loops. The functional logic remains unchanged - only execution scheduling is affected. Standard async patterns applied consistently across the serve/worker subsystem.

^{You can customize Macroscope's approvability policy. Learn more.}

[S1ro1]

lgtm