fix(runners): guard SEGMENT against set -u in start-runner.sh

[jensenbox]

Problem

modules/runners/templates/start-runner.sh assigns SEGMENT only inside the X-Ray branch (line 191) but consumes it unconditionally at line 263 and in the error_handler trap (line 98). With tracing disabled — the default — the branch never executes and SEGMENT is never assigned.

That is harmless while user-data runs without set -u. It becomes fatal the moment anything enables -u earlier in the same shell, and user-data is one concatenated script: ${post_install} is spliced inline at modules/runners/templates/user-data.sh:63. So a consumer whose post_install hook opens with set -euo pipefail — a perfectly ordinary thing for a hook to do — silently arms this latent bug for the module's own script.

Why it is worth fixing rather than documenting

The failure mode is unusually quiet. User-data dies after the runner registers with GitHub but before its listener starts, so:

• the runner appears healthy in the GitHub UI (registered, holds an agent ID);
• it never claims a job;
• scale-down reaps it as idle after the configured window;
• the scaler launches an identically-broken replacement, forever.

From the outside it looks like a capacity or scaling problem, not a script error. We lost every job on one runner pool for ~19 hours before finding it in the instance console:

√ Runner successfully added
√ Settings Saved.
Tagging instance with GitHub runner agent ID: 49534
/var/lib/cloud/instance/scripts/part-001: line 703: SEGMENT: unbound variable
ERROR: runner-start-failed with exit code 1 occurred on 1
FAILED Failed to start cloud-final.service - Cloud-init: Final Stage.

Fix

"$${SEGMENT:-}" at both unguarded call sites.

This restores the script's existing intent rather than changing behaviour. Both helpers already begin with:

local SEGMENT_DOC="$1"
if [ -z "$SEGMENT_DOC" ]; then
  echo "No segment doc provided"
  return
fi

so they were written to tolerate an absent segment doc — the call sites simply never expressed it in a -u-safe way. With tracing enabled nothing changes; with tracing disabled the helpers now hit the early return they already had.

Line 192's echo "$SEGMENT" is deliberately untouched: it sits inside the branch that assigns the variable.

Note the doubled $ — this template is itself rendered through templatefile(), so a bare ${SEGMENT:-} is parsed as a Terraform interpolation and fails the plan with "Extra characters after interpolation expression; Template interpolation doesn't expect a colon at this location". Existing precedent in the same file: $${extra_flags} and $${config} (lines 257, 269).

Verification

• Rendered the template through templatefile() and confirmed the output contains "${SEGMENT:-}" at both sites.
• bash -n over the rendered script.
• Deployed to a production runner pool: instances that previously died at SEGMENT: unbound variable now reach a running listener, claim jobs, and complete them.

Scope

Two characters at two call sites. No behaviour change for tracing-enabled consumers.

[jensenbox]

Still applies cleanly to current main (MERGEABLE, no rebase needed), and still needed — main's start-runner.sh has the unguarded $SEGMENT at both call sites:

 98: create_xray_error_segment "$SEGMENT" "runner-start-failed with exit code $exit_code ..."
263: create_xray_success_segment "$SEGMENT"

Adding real-world impact in case it helps prioritise, since this looks like a trivial two-character change on the surface.

When set -u is in effect and tracing is off, $SEGMENT is unbound, so the trap fires and the runner dies after registering with GitHub but before starting work. The failure is close to invisible: the instance registers, then goes quiet. Nothing appears in the runner logs, because the process is gone before it writes any. The only place the cause shows up is aws ec2 get-console-output.

On our fleet that cost roughly 19 hours of deploy outage before we found it — jobs sat queued against runners that looked healthy in the GitHub UI. We are carrying this patch in a fork to stay unblocked, but it belongs upstream: anyone hitting it has no realistic path to diagnosing it from the logs they would normally check.

Happy to rebase or adjust the form ($${SEGMENT:-} is what the templating needs here) if you would prefer it written differently.