[Bug] worker_task_slots_used reports 1 on an idle worker (oscillates 0↔1) and disagrees with worker_task_slots_available

[Sanil2108]

What are you really trying to do?

Was using this metric for two usecases -

• Observability, this is one of the metrics on our dashboards as a signal for "is this worker actually executing tasks", and utilization as used / (used + available). Both are thrown off because used never settles to 0 on an idle worker.
• Horizontal scaling - We have forked KEDA and have modified the temporal scaler. We calculate the total number of activities in queue + in flight to calculate the right pod count. We use this metric to get the total number of activities in flight.

Describe the bug

On a completely idle worker (no workflow or activity ever scheduled), temporal_worker_task_slots_used for WorkflowWorker and ActivityWorker holds at 1 and periodically drops to 0 and back — a steady 0↔1 oscillation with a period matching the long-poll refresh (~60s). LocalActivityWorker stays at 0. Every transition lines up with an increment of temporal_long_request{operation="PollWorkflowTaskQueue"|"PollActivityTaskQueue"}.

At the same time, worker_task_slots_used and worker_task_slots_available are mutually inconsistent — they don't sum to the configured pool size (100):

temporal_worker_task_slots_used{worker_type="WorkflowWorker"}       1
temporal_worker_task_slots_available{worker_type="WorkflowWorker"}  100   # 1 + 100 = 101 ≠ 100
temporal_worker_task_slots_used{worker_type="ActivityWorker"}       0
temporal_worker_task_slots_available{worker_type="ActivityWorker"}  100
temporal_worker_task_slots_used{worker_type="LocalActivityWorker"}  0

The used=1 is not an executing task — it appears to be the slot permit reserved for the outstanding long-poll (reserve-before-poll). available is not decremented for that same reservation, hence the mismatch. Two facts confirm this:

• A workflow blocked waiting on a running activity shows WorkflowWorker used=1, available=100 even though no workflow task is executing. During that same window the ActivityWorker correctly shows used=1, available=99 (1 + 99 = 100) for the actually-running activity.
• LocalActivityWorker, which is not polled from the server (no reserve-before-poll), never shows the phantom 1.

The metrics reference documents worker_task_slots_used as "the total number of ... Task execution slots in current use", so an idle worker reporting a non-zero value contradicts the documented meaning, and used/available should be complementary for a fixed-size supplier.

Minimal Reproduction

temporalio==1.16.0, default (fixed-size) slot suppliers = 100 each.

# worker.py — idle worker, no workflows/activities ever started
import asyncio
from datetime import timedelta
from temporalio import activity, workflow
from temporalio.client import Client
from temporalio.runtime import Runtime, TelemetryConfig, PrometheusConfig
from temporalio.worker import Worker

@activity.defn
async def say_hello(name: str) -> str:
    return f"Hello, {name}!"

@workflow.defn
class HelloWorkflow:
    @workflow.run
    async def run(self, name: str) -> str:
        return await workflow.execute_activity(
            say_hello, name, start_to_close_timeout=timedelta(seconds=10))

async def main():
    runtime = Runtime(telemetry=TelemetryConfig(
        metrics=PrometheusConfig(bind_address="127.0.0.1:9100")))
    client = await Client.connect("localhost:7233", runtime=runtime)
    async with Worker(client, task_queue="slot-repro-tq",
                      workflows=[HelloWorkflow], activities=[say_hello]):
        while True:
            await asyncio.sleep(3600)

asyncio.run(main())

temporal server start-dev &
python worker.py &
# poll for ~2 min; used flickers 0<->1 while available stays 100:
watch -n0.1 'curl -s localhost:9100/metrics | grep -E "worker_task_slots_(used|available)"'

Environment/Versions

• OS and processor: M1 Mac (arm64), Python 3.11
• Temporal Version: Server 1.27.1 (CLI dev server 1.3.0); SDK: temporalio (Python) 1.16.0 → bundled Rust sdk-core (metrics service_name="temporal-core-sdk")
• Are you using Docker or Kubernetes or building Temporal from source? No — temporal server start-dev + a locally-run Python worker

Additional context

The gauges are emitted by sdk-core, so this should affect the other core-backed SDKs (TypeScript, .NET, Ruby) as well, not just Python.

Two questions for maintainers:

1. Should slots reserved for an in-flight poll count in worker_task_slots_used, given the metric is documented as execution slots in use?
2. If poll reservations are intended to count, why is worker_task_slots_available not decremented in step, so that used + available != pool_size? They appear to come from different accounting.

[Sushisource]

I think this may have been fixed already by #1391 - once we have a release including that, could you close this if it does or comment if it doesn't?