Handle Changes to Slurm's REBOOT workflow

[hintron]

Recent versions of Slurm have made changes in the reboot workflow and added a reboot state. As it is, in Slurm 17.11, doing an scontrol reboot or scontrol reboot asap returns the node back to IDLE before NHC can have a chance to offline the node. This causes jobs to run and potentially fail if a required resource isn't up yet.

I have a preliminary set of patches here that I'm opening up for feedback and suggestions.

I'm still testing things, and I still need to make sure that Slurm 18.08's scontrol reboot nextstate=<STATE> command works with these changes, though I don't believe it will have any affect.

Thanks,
Michael

P.S. This was originally brought up here: https://bugs.schedmd.com/show_bug.cgi?id=6391

[hintron]

What's not clear to me is what versions of Slurm NHC is attempting to target or support. If pre-17.11 Slurm must be supported, then there may need to be some version-specific code.

[hintron]

I'm not yet sure what problem this pull request is attempting to solve, but it seems related: #65

[hintron]

What happens during a reboot in Slurm has also recently been clarified (hopefully to a level that will let NHC know what it can do). See https://slurm.schedmd.com/scontrol.html#OPT_reboot

[hintron]

I'm also not yet sure how #6 relates to all this, either.

[hintron]

In Slurm 17.02, the REBOOT state flag was added along with the ASAP option to scontrol reboot. The internal Slurm node state flags size was increased from 16 bit to 32 bit to accommodate the additional flag. See SchedMD/slurm@861bab6c8ee

[hintron]

Also note that since Slurm 18.08, scontrol reboot accepts nextstate=<RESUME|DOWN>. If this option is used, then Slurm will inevitably clear the DRAIN flag, regardless of what NHC does on the first run after boot up. So basically the nextstate option is not recommended when NHC is in use.

[hintron]

I'm going to clean up a few things and squash the commits into a single commit.Then I'll create a pull request once I think it's ready. The branch will be at https://github.com/hintron/nhc/tree/slurm-node-states.