feat(sdk): auto-enroll default pollers into autoscaling by veeral-patel · Pull Request #1425 · temporalio/sdk-rust

veeral-patel · 2026-07-24T20:19:11Z

Ports temporalio/sdk-go#2442 to sdk-core, so every SDK built on it (Rust directly; TS/Python/.NET/Ruby via the c-bridge) gets it.

Supersedes #1406 — same feature, reworked per review feedback (see "How" below).

What it does

A worker's pollers can run in one of two modes: a fixed number, or autoscaling (the server tells the worker to add/remove pollers based on load).

This makes a worker automatically use autoscaling when the server says the namespace supports it (poller_autoscaling_auto_enroll) — but only for pollers the user didn't configure. If you set a poller count or behavior yourself, it's left exactly as you asked.

How it works

The worker figures out the right mode once, right after it checks in with the server (validate()), and builds its pollers then. Previously the pollers were built before the check-in, so #1406 had to flip the mode later at runtime; this version avoids that entirely (poll_buffer.rs is unchanged here).

worker/mod.rs: poller setup is deferred until after the server check-in, so the mode is known up front.
The Rust SDK now calls validate() on startup (it never did before, so it was missing all namespace capabilities).

Testing

Unit tests for the mode decision and the capability parsing; c-bridge conversion tests. cargo check/fmt/clippy clean; sdk-core lib 368 pass, c-bridge 14 pass.

Sushisource · 2026-07-24T21:50:06Z

+    /// `poller_autoscaling_auto_enroll` capability. Layers above core (the Rust SDK, the c-bridge)
+    /// set this; direct core users leave it `false`.
+    #[builder(default = false)]
+    pub workflow_task_poller_behavior_auto_enroll: bool,


I don't think we need these. They can just explicitly set the option for whatever they do want, rather than disabling the enrollment explicitly.

Good point, I refactored the three poller-behavior fields on WorkerConfig to be Option<PollerBehavior> now

Sushisource · 2026-07-24T21:50:11Z

+impl TryFrom<&PollerBehavior> for Option<temporalio_sdk_core::PollerBehavior> {
    type Error = anyhow::Error;
-    fn try_from(value: &PollerBehavior) -> Result<Self, Self::Error> {
+    fn try_from(value: &PollerBehavior) -> anyhow::Result<Self> {


This won't compile (and doesn't, in CI).

Sushisource · 2026-07-24T21:53:07Z

-    at_task_mgr: Option<WorkerActivityTasks>,
+    /// The poller-dependent subsystems (workflows, activity task manager, nexus manager). Built
+    /// lazily once namespace capabilities are known so effective poller behavior can be resolved.
+    pollers: std::sync::OnceLock<PollerSubsystems>,


These really aren't (just) pollers. TaskSubsystems would be better I think.

Good point, renamed to TaskSubsystems

Sushisource · 2026-07-24T22:07:08Z

+        // If shutdown was already initiated before the pollers were built (e.g. a worker that
+        // never ran validate() and only starts polling as part of draining during shutdown),
+        // propagate the shutdown signal to the freshly-built subsystems so they don't hang waiting
+        // for work that will never arrive. `initiate_shutdown` deliberately does not force a build,
+        // so it can't have done this for us.
+        if shutdown_token.is_cancelled() {


If the task subsystems were never built, we shouldn't need to do anything at all I think?

Good point, removed this unnecessary special case handling

Sushisource · 2026-07-24T22:07:28Z

+    pollers: std::sync::OnceLock<PollerSubsystems>,
+    /// Builds [PollerSubsystems] on first access. Taken exactly once.
+    #[allow(clippy::type_complexity)]
+    poller_builder: parking_lot::Mutex<Option<Box<dyn FnOnce() -> PollerSubsystems + Send>>>,


I think we could avoid needing this, and I think we can kill two birds with one stone and get rid of PollerInitInputs at the same time.

The pollers (now task_subsystems) field, should instead be a https://doc.rust-lang.org/beta/std/sync/struct.LazyLock.html

The closure given to the LazyLock constructor can be the same one you give now, and build_poller_subsystems can be inlined to eliminate the need for PollerInitInputs

Done, turned task_subsystems into a LazyLock. Removed PollerInitInputs and inlined build_poller_subsystems

Sushisource · 2026-07-24T22:11:38Z

+    /// Whether this worker polls for (non-local) activities at all.
+    activities_enabled: bool,


We don't need a separate bool here. Simply not initializing the OnceLock is sufficient.

Done, removed activities_enabled

Sushisource · 2026-07-27T16:41:44Z

+    /// True once `task_subsystems` has been built. Lets shutdown/introspection peek without forcing
+    /// a build (MSRV-safe stand-in for `LazyLock::get`, which is only stable since 1.94).
+    task_subsystems_built: Arc<AtomicBool>,


It's fine for us to upgrade to requiring 1.94 luckily, since Rust SDK is still not GA.

I'm also not sure we actually need to use get though. Since we force loading, it should be fine to just deref the lock where it's used.

Done, dropped LazyLock::get and now just using deref at the use sites

I checked and deref/force/new are all ≤1.80 so should be good to keep MSRV stays at 1.88 for now.

Sushisource · 2026-07-27T16:43:27Z

+    /// namespace capabilities are known so effective poller behavior can be resolved.
+    #[allow(clippy::type_complexity)]
+    task_subsystems:
+        std::sync::LazyLock<TaskSubsystems, Box<dyn FnOnce() -> TaskSubsystems + Send>>,


Nit: remove the std::sync:: prefix and just import directly.

Done, imported LazyLock/OnceLock directly and dropped the std::sync:: prefixes

Sushisource · 2026-07-27T16:48:52Z

+        // can be resolved with knowledge of those capabilities. Everything it needs is captured by
+        // clone (or moved) *before* the synchronous heartbeat manager below consumes the originals.
+        let task_subsystems_builder: Box<dyn FnOnce() -> TaskSubsystems + Send> = {
+            let task_subsystems_built = task_subsystems_built.clone();


Fairly sure not all of these need to actually be clone()d and some of them could just be moved into the closure.

Looked into this, tried removing the clone calls but it causes compiler errors. Unfortunately won't be possible in this case but thanks for flagging!

When a namespace advertises the `poller_autoscaling_auto_enroll` capability, workers automatically switch workflow, activity, and nexus pollers to autoscaling — but only for poller types the user left at their default (neither a fixed poller count nor an explicit behavior). Explicitly configured pollers are left unchanged. The effective `PollerBehavior` is resolved once, after the namespace capability is known. Poller construction is deferred out of the synchronous `Worker::new_with_pollers` into a lazily-built step that runs after `Worker::validate()` populates capabilities (and on first poll for paths that skip validate, e.g. replay). This keeps `poll_buffer.rs` / `PollScaler` free of any runtime mode-switching — the behavior is simply passed in already resolved. Eligibility (user left the poller at its default) is signalled from the layers above core: the Rust SDK exposes the poller options as `Option`, and the c-bridge treats unset (both FFI pointers null) as eligible. Ports temporalio/sdk-go#2442.

Worker::run() now calls validate(), which describes the namespace to discover capabilities. Mock-based tests never wired up that expectation, so mockall panicked; during unwind this became a non-unwinding abort that killed the whole test process and cascaded into unrelated tests. Add a fallback describe_namespace returning an empty response (no capabilities, so poller behavior stays at its default) for both the automock (build_mock_pollers) and manual (mock_manual_worker_client, used by replay) paths. Registered with an open call count and after any test-provided expectation, so mockall's FIFO matching still prefers a test's own expectation and workers that never validate don't trip an unsatisfied expectation.

veeral-patel · 2026-07-29T19:21:05Z

I manually tested this PR locally in all 5 Rust-based SDKs (Rust, Python, .NET, Ruby, TypeScript) successfully.

Steps:

I cloned each SDK locally and modified it to point to my local sdk-rust (core) repo with these changes
Started local Temporal server, with frontend.pollerAutoscalingAutoEnroll on and then off (confirmed via grpcurl)
Added temporary eprintln! line in resolve_effective_behavior (in sdk-rust) to print the resolved poller behavior (fixed or auto)

Result — ✅ all five pass

| SDK | 7233 (OFF) | 7234 (ON) |
|---|---|---|
| Rust / Python / TypeScript / Ruby / .NET | SimpleMaximum(5) | Autoscaling{1,100,5} |

Pending: We need to upgrade the Rust core version in all the Rust based SDKs. SDK team will handle this (discussed with Spencer)

veeral-patel requested a review from a team as a code owner July 24, 2026 20:19

veeral-patel force-pushed the feat/poller-autoscaling-auto-enroll-v2 branch 2 times, most recently from 5954b11 to d600242 Compare July 24, 2026 21:22

veeral-patel mentioned this pull request Jul 24, 2026

Auto-enroll pollers into autoscaling on PollerAutoscalingAutoEnroll (Rust SDK) #1406

Closed

Sushisource reviewed Jul 24, 2026

View reviewed changes

veeral-patel force-pushed the feat/poller-autoscaling-auto-enroll-v2 branch from d600242 to e32bf5f Compare July 25, 2026 00:46

Sushisource reviewed Jul 27, 2026

View reviewed changes

veeral-patel force-pushed the feat/poller-autoscaling-auto-enroll-v2 branch 2 times, most recently from 8489fd7 to 8611a23 Compare July 27, 2026 21:03

Sushisource approved these changes Jul 27, 2026

View reviewed changes

veeral-patel and others added 5 commits July 27, 2026 16:53

Reduce clones somewhat

5f3cd12

Add regression for validation race

721578e

Ensure various calls to describe namespace are synchronized

1a84794

Sushisource force-pushed the feat/poller-autoscaling-auto-enroll-v2 branch from cecd2e3 to 1a84794 Compare July 27, 2026 23:57

Sushisource merged commit 3133db0 into main Jul 28, 2026
22 checks passed

Sushisource deleted the feat/poller-autoscaling-auto-enroll-v2 branch July 28, 2026 18:50

		/// Whether this worker polls for (non-local) activities at all.
		activities_enabled: bool,

Conversation

veeral-patel commented Jul 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What it does

How it works

Testing

Uh oh!

Choose a reason for hiding this comment

Uh oh!

veeral-patel Jul 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

veeral-patel Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

veeral-patel Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

veeral-patel commented Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

veeral-patel commented Jul 24, 2026 •

edited

Loading

veeral-patel Jul 26, 2026 •

edited

Loading

veeral-patel Jul 27, 2026 •

edited

Loading

veeral-patel Jul 27, 2026 •

edited

Loading

veeral-patel commented Jul 29, 2026 •

edited

Loading