validation should fail when setting per_alloc on a dynamic host volume

[inahga]

Nomad version

Output from nomad version

Nomad v1.10.4
BuildDate 2025-08-12T20:48:32Z
Revision 62b195aaa535b2159d215eaf89e6f4a455d6f686

Operating system and Environment details

Ubuntu 24.04 on QEMU/KVM.

Issue

When using dynamic host volumes, you can use the per_alloc setting.

This contradicts the documentation, which states:

Use per_alloc only with CSI volumes and sticky only with dynamic host volumes.

If this is the case, a host volume with per_alloc = true should fail job validation.

Reproduction steps

Create a host volume

name      = "example[0]"
type      = "host"

plugin_id = "mkdir"

capability {
  access_mode     = "single-node-writer"
  attachment_mode = "file-system"
}

Deploy a spec

job "example" {
  type      = "service"
  node_pool = "ifn"

  group "ubuntu" {
    count = 1

    volume "data" {
      type   = "host"
      source = "example"
      per_alloc = true
    }

    task "ubuntu" {
      driver = "podman"

      config {
        image = "docker.io/library/ubuntu:noble"
        command = "/usr/bin/sleep"
        args = ["infinity"]
      }

      volume_mount {
        volume      = "data"
        destination = "${NOMAD_ALLOC_DIR}/data"
      }
    }
  }
}

Expected Result

Actually, this behavior is totally fine! I like it this way, since it solves the problem of allocating multiple host volumes to a replicated service (e.g. mysql, etcd).

I'm just not sure if this was the intended behavior behind per_alloc, if the documentation is wrong, and/or whether using per_alloc in this way has unforeseen consequences and risks.

Can the maintainers tell me whether this is a broken usage of per_alloc. If so, I'd actually like us to fix the problems that prevent it from being used for host volumes (happy to help). If not, we should update the documentation to state using per_alloc is OK.

Actual Result

It deploys fine (hopefully a good thing!)

Job file (if appropriate)

Nomad Server logs (if appropriate)

Nomad Client logs (if appropriate)

[tgross]

Edit: ignore everything I wrote originally here, as I was misreading your post. Fixed version below: 😊

Hi @inahga! You're right, this should fail validation. I'll mark this for roadmapping.

[inahga]

More broadly, what can we do to make per_alloc work for dynamic host volumes? What about it breaks when applied to DHVs and not CSI volumes?

I think its behavior is desirable for DHVs. In fact, it is probably a blocker for us using DHVs and running stateful workloads on Nomad. The example scenario I outlined in the issue is actually what I want to happen, and it ostensibly works, but it's relying on undocumented and maybe subtly broken behavior.

[inahga]

And perhaps to prevent XY problem: I want to use DHVs to run stateful workloads that do replication at the application layer, e.g. let's just say mysql. So having host-attached volumes is totally fine.

The problem with a naive approach (i.e. without per_alloc) is that I can't make a mysql group where each replica has its own volume.

The best I can do is dynamic "group", like so.

variables {
  volume_index = ["0", "1", "2", "3"]
}

job "mysql" {
  type      = "service"

  dynamic "group" {
    for_each = var.volume_index
    labels   = ["mysql${group.value}"]

    content {
      volume "mysql" {
        type   = "host"
        source = "mysql${group.value}"
        sticky = true
      }
      task "mysql" {
        // ...
        volume_mount {
          volume      = "mysql"
          destination = "${NOMAD_ALLOC_DIR}/data"
        }
      }
    }
  }

  update {
    max_parallel     = 1
    auto_revert      = true
    healthy_deadline = "9m"
    min_healthy_time = "30s"
  }
}

This works but not well. Updates don't really work in this scenario--if I start a new job, then each independent group is restarted at the same time, causing downtime.

The other alternative is to define a job for each volume, but then I'm on my own for scheduling updates. This defeats one of the key uses for Nomad.

I think this saga has been played out before for CSI volumes. See #7877