api: add numNodes spec

[jgehrcke]

Add specification for current numNodes behavior as well as guidance towards non-reliance on the CD-global Status field.

As we remove this soon, it doesn't need to be perfect. But I'd like for us to get into the habit of having complete and correct specification of all parameters in our API surface; that will then be the reference for code -- and spec patches are some of the most interesting ones.

Resolves #615.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[klueska]

We should only recommend setting numNodes to 0 when IMEXDaemonsWithDNSNames=true. Setting it to 0 when IMEXDaemonsWithDNSNames=false will result in IMEX daemons being brought online without first filling out their nodes_config.cfg files with all indended IPs of the ComputeDomain. That was the whole point of the gating in the first place.

[jgehrcke]

only recommend setting numNodes to 0 when IMEXDaemonsWithDNSNames=true.

Of course. Will make an update.

This really is clear to me -- the fact that I forgot to make that case distinction explicitly is a reflection of how enthusiastic I am about leaving the "legacy world" behind us ASAP: precisely to not have to think through everything from two very different perspectives for a long period of time.

[klueska]

Probably also worth saying that it must be equal to the expected number of workers joining the ComputeDomain when IMEXDaemonsWithDNSNames=false

[jgehrcke]

I have pushed another update. I like this discussion + iteration.