Secret block with Nomad provider causes high-frequency API calls when WorkloadID permissions are insufficient

[mhehle]

Nomad version

Nomad v1.11.0
BuildDate 2025-11-11T16:18:19Z
Revision 9103d93

Operating system and Environment details

Linux 6.6.107-1-MANJARO

Issue

When using the new secret block in a job specification, Nomad repeatedly issues GET /v1/var/... requests to its own API if the WorkloadID does not have sufficient ACL permissions for the referenced variable path.

Reproduction steps

1. Enable ACLs in Nomad
2. Create a variable e.g.: secrets/abc1.
3. Define a job using the secret block with provider nomad, referencing secrets/abc1.
4. Deploy the job
5. Observe logs: Nomad repeatedly sends GET /v1/var/secrets/abc1 requests, all returning 403 Permission denied.

Expected Result

• proper backoff strategy

Actual Result

Job file (if appropriate)

Nomad Server logs (if appropriate)

    2025-11-17T14:00:15.285+0100 [DEBUG] http: request complete: method=GET path="/v1/var/secrets/abc1?index=1&namespace=my-namespace&stale=&wait=300000ms" duration="182.481µs"
    2025-11-17T14:00:15.388+0100 [DEBUG] http: request failed: method=GET path="/v1/var/secrets/abc1?index=1&namespace=my-namespace&stale=&wait=300000ms" error="Permission denied" code=403
    2025-11-17T14:00:15.388+0100 [DEBUG] http: request complete: method=GET path="/v1/var/secrets/abc1?index=1&namespace=my-namespace&stale=&wait=300000ms" duration="189.02µs"
    2025-11-17T14:00:15.501+0100 [DEBUG] http: request failed: method=GET path="/v1/var/secrets/abc1?index=1&namespace=my-namespace&stale=&wait=300000ms" error="Permission denied" code=403
    2025-11-17T14:00:15.501+0100 [DEBUG] http: request complete: method=GET path="/v1/var/secrets/abc1?index=1&namespace=my-namespace&stale=&wait=300000ms" duration="226.391µs"
    2025-11-17T14:00:15.615+0100 [DEBUG] http: request failed: method=GET path="/v1/var/secrets/abc1?index=1&namespace=my-namespace&stale=&wait=300000ms" error="Permission denied" code=403
    2025-11-17T14:00:15.615+0100 [DEBUG] http: request complete: method=GET path="/v1/var/secrets/abc1?index=1&namespace=my-namespace&stale=&wait=300000ms" duration="191.09µs"
    2025-11-17T14:00:15.726+0100 [DEBUG] http: request failed: method=GET path="/v1/var/secrets/abc1?index=1&namespace=my-namespace&stale=&wait=300000ms" error="Permission denied" code=403
    2025-11-17T14:00:15.726+0100 [DEBUG] http: request complete: method=GET path="/v1/var/secrets/abc1?index=1&namespace=my-namespace&stale=&wait=300000ms" duration="190.92µs"
    2025-11-17T14:00:15.839+0100 [DEBUG] http: request failed: method=GET path="/v1/var/secrets/abc1?index=1&namespace=my-namespace&stale=&wait=300000ms" error="Permission denied" code=403
    2025-11-17T14:00:15.839+0100 [DEBUG] http: request complete: method=GET path="/v1/var/secrets/abc1?index=1&namespace=my-namespace&stale=&wait=300000ms" duration="190.24µs"
    2025-11-17T14:00:15.952+0100 [DEBUG] http: request failed: method=GET path="/v1/var/secrets/abc1?index=1&namespace=my-namespace&stale=&wait=300000ms" error="Permission denied" code=403
    2025-11-17T14:00:15.952+0100 [DEBUG] http: request complete: method=GET path="/v1/var/secrets/abc1?index=1&namespace=my-namespace&stale=&wait=300000ms" duration="181.68µs"

Nomad Client logs (if appropriate)