perf(orchestrator): reduce event-loop lag

[mikasenghaas]

Summary

Reduce orchestrator step-boundary event-loop lag, motivated by router-replay runs (Qwen3-30B-A3B + GLM-5.1) where the per-step routed_experts payload (~25 GB across 256 rollouts) caused multi-second loop-lag spikes at step boundaries.

Changes are all in the orchestrator and its post-rollout pipeline. The trainer/inference paths are untouched.

Coupled with PrimeIntellect-ai/verifiers#1453 (perf/r3-serve). The verifiers PR fixes the env-side counterpart of the same lag (env_worker / env_router), which this PR's verifiers submodule bump pulls in. Land or revert together — without the verifiers side the env worker loop saturates first under 256-rollout concurrency and the orch-side gains here don't surface.

Follow-up #2612 (perf/r3-sidecar) layers a routed_experts sidecar-split sender on top of this PR. Use it if the to_thread-only send here isn't enough for the largest router-replay payloads.

What's in here

7 commits, ~200 LOC. Each commit is a single intervention:

#	Commit	What
1	perf(transport): offload TrainingBatch send to a worker thread	Make TrainingBatchSender.send async; encode + disk write run via asyncio.to_thread so the orch event loop stays responsive during step transitions.
2	perf(orchestrator): await async sender; to_thread advantages+filters+uvloop	Await the now-async sender. Wrap compute_advantages and apply_filters in asyncio.to_thread — pure-Python work that was running on the loop. Install uvloop in main().
3	perf(orchestrator): log p99 event-loop lag in metrics + busy-loop warning	Add event_loop_lag/p99 to the wandb dict and the busy-loop warning line. Raise warn thresholds to p90>1s / p99>5s / max>30s — p90 alone undersold the long tail we actually care about.
4	perf(orchestrator): defer routed_experts concat in interleave_rollout	Per-extension unpack → numpy.concatenate → repack was O(N²) byte copies (~2.3 GB memcpy per long rollout). New path keeps a per-sample chunk list and concatenates once at finalize. O(N) byte work, ~100× less per rollout.
5	perf(orchestrator): orjson save_rollouts + dedup bool conversion	json.dump → orjson.dumps. Pure-Python serializer held the GIL for the whole save (multi-second spikes on big steps); orjson serializes in C and releases the GIL. Also dropped the redundant [bool(i) for i in mask] re-conversion in make_sample/extend_sample (use list(...) instead) and switched prepare_step_tokens to list(map(bool, ...)).
6	perf(orchestrator): skip pretokenize fanout when nothing needs work; 10Hz lag monitor	Pretokenize is a no-op for router-replay (every step has tokens populated), but the 256-way to_thread fanout was firing only to no-op and the GIL stampede blocked the loop. Skip when no rollout needs work. Also EventLoopLagMonitor sample rate 1 Hz → 10 Hz to catch sub-second spikes.
7	chore(deps): bump verifiers submodule to PR 1453 head	Pulls in the env_worker / env_router lag fixes from verifiers#1453.

What's NOT in here (deferred / dropped)

• Streaming + routed_experts sidecar in the sender — moved to follow-up perf(transport): stream-sidecar TrainingBatch sender (routed_experts split) #2612. Real win if the simpler to_thread send here is insufficient under heavy router replay, but adds an on-disk format change and per-sample loop yields.
• Shared ProcessPoolExecutor — tried for router-replay's GIL residual, didn't help (per-call pickle/IPC cost dominated the small per-rollout payload). Dropped.
• Per-phase timing instrumentation + dump_raw_rollouts — useful while investigating but not worth keeping in prod.
• Chunked gather (gather_chunk_size) — sliced interleave_rollout waves with await sleep(0) between. Real wallclock cost for unclear value once the algorithmic fixes (Data streaming #4) landed. Reverted.
• Killing base64 on the routed_experts wire — profile shows pybase64.b64decode_as_bytearray is ~73% of single-thread CPU in interleave_rollout. Releases the GIL so it parallelises and doesn't block the loop directly, but it's the dominant wallclock floor. Moving inference→orch transport to msgpack bin would eliminate the encode+decode. Cross-repo, deferred.
• TrainingSample.{prompt,completion}_mask: list[bool] → list[int] or bytes — would kill the last Python loop in prepare_step_tokens. Interface change, deferred until empirical evidence it matters.

---

Note

Medium Risk
Touches the orchestrator post-rollout pipeline and transport TrainingBatchSender API, so regressions could impact training throughput or batch delivery ordering, though changes are largely performance-oriented and isolated from model/inference logic.

Overview
Reduces orchestrator event-loop stalls during step transitions by pushing CPU/GIL-heavy work off the loop and cutting per-rollout copying overhead.

compute_advantages, rollout filtering, and filesystem batch encoding/writes now run via asyncio.to_thread, and TrainingBatchSender.send becomes async (with the orchestrator awaiting it). Pretokenization fanout is skipped when all steps already have tokens, and uvloop is installed in main().

interleave_rollout now defers routed_experts concatenation/packing until finalization (chunking per step and concatenating once), and rollout JSONL saving switches to orjson. Event-loop lag monitoring is sampled at 10Hz and logs/exports p99 lag with updated warning thresholds.

^{Reviewed by Cursor Bugbot for commit b92f7d3. Bugbot is set up for automated code reviews on this repo. Configure here.}

[S1ro1]

love