rfc44: support jobid and possibly other fields in drain events

[chu11]

There are several issues related to reporting errors to users by looking at non-job specific data:

flux-framework/flux-core#7135
flux-framework/flux-core#7136

We would like to parse resource.eventlog to associate a drain event with a specific job, however the current lack of extra drain information makes this minimally inconvenient, but likely very hard when a number of corner cases are thrown in.

Example, lets say that housekeeping fails and drains a node. A user or admin may want to associate the housekeeping failure with the job that just ran before it. In order to do that we would have to 1) see what time the node was drained 2) find the last job that finished on that node before that timestamp.

But this is the algorithm only if we apriori know housekeeping drained the node. The algorithm for associating a drain event from epilog is different. Also, nothing indicates where the drain event may have come from. What if an admin drains a node for maintenance after a job runs, it could appear it came from housekeeping.

I think we should have additional information in the drain event when that information is available.

• jobid - optional field could tie drain event to a specific jobid. This could be automatic in flux resource drain if the FLUX_JOB_ID environment variable exists.

• reporter - could be housekeeping, epilog, "user" (command line). Like if an admin drains a node for some very specific purpose, we'll know it wasn't related to a job per se.

Maybe some other info could be useful. Only bits I thought of so far.

[garlick]

Just to be clear about the current state of drain reasons when flux drains a node

• If housekeeping fails, the housekeeping unit file drains the node with a reason containing "housekeeping@jobid", but if the node is already drained, e.g. by the site provided scripts, then the site reason is not overwritten.
• If prolog/epilog fails, the job manager perilog plugin drains the node with the reason "{prolog|epilog} {canceled then timed out|timed out|failed} for job jobid" but again won't overwrite an existing drain message
• If there is a job exec failure, reasons might be "job jobid start timeout: possible node hang", "sdexec reports error-string for job jobid", or "jobid terminated before first barrier",

Seems like a good idea to me to add the jobid and reporter (mabye use the shorter string who?) as OPTIONAL (for backwards compatibility) structured data.

Having flux resource drain include the jobid automatically would address the problem of site-provided scripts performing the drain without requiring any action from them. Maybe there is some way to determine some context as well for reporter/who.