Fix two chain-killing bugs in the PBS self-scheduling recipe

docs/nightly-smoke-testing.md

@@ -56,7 +56,8 @@ Two recipes with placeholders:
 
 3. **Verify a full cycle.** After the first run finishes, confirm:
    - a next run is queued — `squeue --me -n rootstock-nightly` (SLURM) /
-     `qstat -u $USER` (PBS) shows a pending/queued job, and
+     `qstat -u $USER` (PBS) shows a pending job (on PBS a job with a future
+     start time sits in state `W`, waiting — not `Q`), and
    - the manifest landed — `rootstock status` shows fresh `verified_at` times.
 
 ## How the self-scheduling works
@@ -85,10 +86,14 @@ Remove the stop file to allow a fresh kickoff later.
 ### Testing without waiting a week
 
 ```bash
-RESCHEDULE_BEGIN=now+3minutes sbatch scripts/nightly_smoke_test.sbatch   # SLURM
-qsub -a $(date -d "+3 minutes" +%m%d%H%M) scripts/nightly_smoke_test.pbs  # PBS
+RESCHEDULE_BEGIN=now+3minutes sbatch scripts/nightly_smoke_test.sbatch                    # SLURM
+qsub -v RESCHEDULE_AT=$(date -d "+3 minutes" +%m%d%H%M) scripts/nightly_smoke_test.pbs    # PBS
 ```
 
+Both override when generation 1 schedules generation 2, which is the handoff
+worth testing. (On PBS, `qsub -a <time>` would only delay generation 1's own
+start and exercises nothing about the chain.)
+
 ## Knobs
 
 Override via env at submit time (SLURM: inline `VAR=val sbatch …`; PBS:

scripts/nightly_smoke_test.pbs

@@ -17,19 +17,27 @@
 # The push IS the result channel, so nothing off-cluster waits on this job.
 #
 # One-time kickoff (submit from the directory containing this script; PBS has
-# no reliable way to recover the script path mid-job, so we reference it by
-# its submit directory):
+# no reliable way to recover the script path mid-job, so the FIRST generation
+# derives it from its submit directory, and every generation after that is
+# handed it explicitly via `qsub -v SELF_PATH=...`):
 #   qsub scripts/nightly_smoke_test.pbs
 #
 # Stop the chain:
 #   touch ~/.rootstock-nightly-stop      # next run won't reschedule
-#   qstat -u $USER                       # find the queued (Q) next run
+#   qstat -u $USER                       # find the waiting (W) next run
 #   qdel <jobid>                         # delete it
+# (A job submitted with -a sits in state W, not Q, until its start time.)
 #
-# Fast end-to-end test (reschedule ~3 min out instead of next week):
-#   qsub -a $(date -d "+3 minutes" +%m%d%H%M) scripts/nightly_smoke_test.pbs
-# (passing -a on the command line just kicks off the chain sooner; subsequent
-#  runs use the cadence below.)
+# If you MOVE or RENAME this script: the waiting next job carries the old
+# path, so qdel it and resubmit once from the new location.
+#
+# Fast end-to-end test (reschedule ~3 min out instead of next week, so you can
+# watch generation 1 hand off to generation 2):
+#   qsub -v RESCHEDULE_AT=$(date -d "+3 minutes" +%m%d%H%M) scripts/nightly_smoke_test.pbs
+# (Overriding RESCHEDULE_AT exercises the real handoff. `qsub -a <time>` would
+#  only delay generation 1's start and tests nothing about the chain. Knob
+#  overrides apply to generation 1 only — except CLUSTER, which is passed down
+#  the chain; the in-script defaults govern every subsequent generation.)
 
 set -euo pipefail
 
@@ -50,22 +58,30 @@ RESCHEDULE_AT="${RESCHEDULE_AT:-$(date -d "+${CADENCE_DAYS} days ${RUN_HOUR}" +%
 STOP_FILE="${STOP_FILE:-$HOME/.rootstock-nightly-stop}"
 JOB_NAME="${JOB_NAME:-rootstock-nightly}"
 
-# PBS copies the script to a spool dir, so $0 is unreliable. Reference the
-# original by submit directory (override SELF_PATH if you submit from elsewhere).
+# PBS copies the submitted script to a spool dir, so $0 is unreliable. On the
+# FIRST submission SELF_PATH is derived from the submit directory
+# ($PBS_O_WORKDIR); every generation then passes it down explicitly via
+# `qsub -v` (below) — a rescheduled job's own $PBS_O_WORKDIR is useless, it's
+# just the parent job's cwd ($HOME, since PBS starts jobs there).
 SELF_PATH="${SELF_PATH:-${PBS_O_WORKDIR:-$(pwd)}/nightly_smoke_test.pbs}"
 
 # --- Reschedule FIRST -------------------------------------------------------
 # Queue the next run before any work, so a crash in the smoke-test can't break
-# the chain. The next job sits queued (Q) until -a; qdel it to stop the chain.
+# the chain. The next job sits waiting (W) until -a; qdel it to stop the chain.
 if [[ -z "${PBS_JOBID:-}" ]]; then
   echo "not running under PBS; skipping reschedule (dry run)."
 elif [[ -e "$STOP_FILE" ]]; then
   echo "stop file present ($STOP_FILE); chain ends here."
-elif qselect -u "$USER" -N "$JOB_NAME" -s Q 2>/dev/null | grep -q .; then
-  # Belt-and-suspenders against duplicate chains.
-  echo "a '$JOB_NAME' job is already queued; not scheduling another."
+elif qselect -u "$USER" -N "$JOB_NAME" -s QW 2>/dev/null | grep -q .; then
+  # Belt-and-suspenders against duplicate chains. QW: a job with a future
+  # start time (-a) sits in state W, not Q.
+  echo "a '$JOB_NAME' job is already queued/waiting; not scheduling another."
 else
-  next_id="$(qsub -a "$RESCHEDULE_AT" "$SELF_PATH")"
+  # -v hands the next generation its own location (PBS does not propagate the
+  # environment down the chain) plus CLUSTER, which must survive to every
+  # generation on shared installs. -v passes ONLY the listed variables, so
+  # other knob overrides still die with generation 1, as before.
+  next_id="$(qsub -a "$RESCHEDULE_AT" -v SELF_PATH="$SELF_PATH",CLUSTER="$CLUSTER" "$SELF_PATH")"
   echo "scheduled next run: job $next_id (begins at $RESCHEDULE_AT)"
 fi