Dynamic Host Volumes Deferred Binding

[ahzay]

Proposal

Dynamic Host Volumes (#15489) could allow for deferred binding (similar to CSI create/publish distinction) by adding an optional flag in the fingerprint. If the plugin's fingerprint.deferred_bind (or similar) is enabled, then the create/delete RPC calls to the DHV plugin would only happen during the task pre/post hooks. "Creating" the volume in nomad would only store it in the state, and not actually call the plugin binary.

Use-cases

Dynamic Host Volumes currently always bind to a node during create. This is not suitable for plugins backed by remote storage. Binding during create makes it pretty much useless to build those kinds of network-backed plugins, for which volumes have no reason to be bound to any specific node until scheduling time. I find this to be a very desirable application for DHV because it is much simpler than CSI, it would be much easier to maintain a wide variety of plugins, networked or otherwise. Also, having this deferred bind "mode" hard-coded at the fingerprint level means simpler plugin code, a plugin dev could have two binaries, one for each mode.

Attempted Solutions

Have not attempted a solution yet. I have been looking at the codebase to figure out where the best place would be to insert these deferred calls, so far I'm thinking around somewhere in the task hooks, perhaps creating a new hook similar to the csi_hook. I would appreciate any input from people more familiar with it.

[tgross]

@ahzay the problem with deferred binding, as you'll find if you look at the very long history of CSI bug fixes 😁 , is that the scheduler needs to be able to know if it can place an allocation on a given node and with a given volume's access_mode before the allocation is placed.

So it's not just a matter of mounting during the allocation hooks (although yes, I think that's where you'd do that step). You've also got to figure out all the logic around how to do feasibility checking. You'd also probably want to figure out a plan for automatic unbinding once the volume is done or if the task fails for unrelated reasons. i.e. if we mount the volume and then the task crashes and can't restart, do we keep the volume there forever? If we do unmount it, how do we handle lost nodes where the node drops out and never comes back? This is where a lot of the complexity of CSI comes from, so that's why the deferred binding wasn't a feature of DHV.

The current architecture would also trigger the node fingerprint to get updated and unblock any evaluations waiting on that node when the allocation hook fires. That's a minor performance issue and probably not anything to worry about, but just another though.

So I'm not saying that this shouldn't be done, but it's got a lot of hidden surface area. 😁

[ahzay]

@tgross Thanks for the response, I did figure that this is quite complex and would need more discussion. I do still think it can be done while avoiding a lot of the CSI complexity, I'll give me two cents while I try to figure out how to implement this:

You've also got to figure out all the logic around how to do feasibility checking

the node needs the plugin and the free space, right?

You'd also probably want to figure out a plan for automatic unbinding once the volume is done or if the task fails for unrelated reasons. i.e. if we mount the volume and then the task crashes and can't restart, do we keep the volume there forever?

This is where the main difference can be, CSI (the controller) assumes the responsibility for coordinating the mounts, DHV could avoid that. Also, from what I've seen the CSI publish op is run when creating the task dir, and unpublish is run when destroying it, not at alloc time (correct me if I'm wrong). The DHV could have the same behavior.

If we do unmount it, how do we handle lost nodes where the node drops out and never comes back?

Assuming that the mount coordination is handled elsewhere, we can simply run the mount operation again on another node upon creating the task directory on that new node. If the old node ever comes back, assuming the plugin operations are idempotent, we can simply run an unmount on the old node as soon as it comes back to make sure there's no remaining mount process left stale (the plugin is responsible for this cleanup).

So that's the two assumptions I'd work with:

• nomad's DHV is not in charge of coordinating mounts, if the mount is impossible (because of a duplicate network mount for example) the plugin call should just fail, it's the responsibility of the plugin. However it should probably be able to detect stales and mount successfully for network mounts, so moving the task to another node succeeds.
• DHV plugin operations should be idempotent, so trying to unmount a non-existent mount should not error out, nor should trying to mount a volume that is already mounted on the present node. Running the DHV's unmount should leave no orphaned process related to the mount.

Of course, I understand that all of this is up to the discretion of the maintainers. I am just very interested in having this discussion if you don't mind 😃

[tgross]

the node needs the plugin and the free space, right?

And the mode, so that it can schedule read-only or exclusively for a given allocation.

This is where the main difference can be, CSI (the controller) assumes the responsibility for coordinating the mounts, DHV could avoid that. Also, from what I've seen the CSI publish op is run when creating the task dir, and unpublish is run when destroying it, not at alloc time (correct me if I'm wrong).

Close. There's a split between the controller and the node plugin RPCs. The CSI spec leaves the behavior of various steps like "publish" implementation-defined by the plugins. But in our observations, but the plugins have converged on a set of typical workflows:

• The Controller Publish step typically "attaches" the volume -- performs the storage provider API calls that cause the device to appear on the host. For example, in AWS, attaching the AWS EBS volume to a particular AWS EC2 instance. For something like NFS this step might include mounting the volume, or that might come in the Node Stage step. We don't have control over that for CSI.
• The Node Stage step will idempotently ready the volume for use (ex. formatting it) and mount the device to the plugin's workspace
• The Node Publish step will bind-mount the volume from the plugin's workspace to the target workspace (allocation directory)
• (and then, in Nomad, the task driver will bind-mount the volume from the allocation directory to wherever you've specified volume_mount)

If the old node ever comes back, assuming the plugin operations are idempotent, we can simply run an unmount on the old node as soon as it comes back to make sure there's no remaining mount process left stale (the plugin is responsible for this cleanup).

For network-attached storage that's a recipe for data corruption unless the job also has disconnect.stop_on_client_after to ensure it's not writing to a volume from two different allocations. For storage like EBS volumes, you may not be able to recover at all without sending the appropriate APIs to the storage provider.

nomad's DHV is not in charge of coordinating mounts, if the mount is impossible (because of a duplicate network mount for example) the plugin call should just fail, it's the responsibility of the plugin. However it should probably be able to detect stales and mount successfully for network mounts, so moving the task to another node succeeds.

We generally try to avoid a workflow where the scheduler could know that an allocation cannot be placed but doesn't know and tries anyways (every place we don't do this, like task.config block validation, we regret it). This also ends up getting expensive to get to that point where idempotency protects us: the scheduler has had to run, a plan applied, allocations created, the allocation has placed, networking has been configured, etc. and then it fails. At high volumes of work this causes a lot of cluster churn. In this case you'd be opting in to the behavior so maybe that's not as big a deal (so long as it came with a pile of warnings in the docs). But even in that case the client should be enforcing the access_mode restrictions, otherwise we're relying on the plugins for correctness of a Nomad jobspec value. That feels like it would be surprising behavior.

I am just very interested in having this discussion if you don't mind

For sure! It helps work through the design constaints. I think this conversation is making me realize that this sort of behavior would probably need to be opt-in for the jobspec author, not just the volume author. Static and dynamic host volumes have identical behaviors to the job author today. But deferred binding has implications for the job author that they need to know about.

[ahzay]

Before bouncing off your comment, let me just mention a big plus that DHV offers: being able to patch the plugin binary without downtime. Uptime is the reason I was thinking about replacing CSI in the first place, I forgot to mention it in the OP.

Close. There's a split between the controller and the node plugin RPCs.

That is true, I was aware but totally forgot because with my drivers I always tend to do it in a way where the controller ops don't do anything and the mounting logic happens with node publish/unpublish. I was very happy to see nomad having the "monolith" option for CSI, it made sense to me. That's also why I suggested deferring the DHV mount instead of adding the same CSI distinction and additional operations.

For network-attached storage that's a recipe for data corruption unless the job also has disconnect.stop_on_client_after

I did not know about this! this is very cool and forcing this implicitly when deferring could come in useful.

We generally try to avoid a workflow where the scheduler could know that an allocation cannot be placed but doesn't know and tries anyways

That is true. There's actually no reason to put the plugin in charge of coordinating when nomad could do it, do you agree on that? That idea was really just my CSI trauma speaking I don't think I'd do it like that anymore 😆

deferred binding has implications for the job author that they need to know about

That makes sense to me, especially if there's a whole host of behaviors it is forcing. I could just be an explicitly required flag in the job spec where planning would fail if it is set and the plugin is not of the deferred type (or vice-versa).