[PRODENG-3633] Add overall apply/reset deadline and surface retry errors

[james-nesbitt]

What

Add an overall deadline for apply/reset, and surface real errors during the MKE health-check retry loop instead of a repeated generic message.

Why

PRODENG-3633: today there is no overall deadline on apply/reset, so any wait that sits outside a bounded retry can block forever with no output and no failure. The specific bug that originally triggered this (pingHost's WaitGroup.Wait() deadlock) is already fixed under PRODENG-3594; this ticket is about the fact that nothing would catch the next one.

How

• phase.Manager gains a Deadline time.Duration field. Run races each phase against it in a goroutine; on timeout it returns an error naming the in-progress phase and does not start the next one. This is a Manager-level wrapper (Option A), not a context.Context threaded through all 30+ phase implementations (Option B) — smaller diff, doesn't make individual waits cancellable, but launchpad is a short-lived CLI so the abandoned goroutine is reclaimed when the process exits after returning the error.
• New Product.SetTimeout(time.Duration) method rather than a parameter on Apply/Reset, so none of the existing callers (cmd/apply.go, cmd/reset.go, test/integration, test/smoke/*) need signature changes.
• New --timeout flag on apply/reset, default 90m (matches the longest existing smoke test timeout in the Makefile), --timeout 0 disables it.
• pingHost's retry loop now logs the real failure via retry.OnRetry on every attempt instead of only "waiting for MKE ... to become healthy" repeated with no reason. The final returned error already carried every attempt's error via retry-go's own Error type; nothing was being discarded, it just wasn't visible while waiting.
• Audited every other sync.WaitGroup use in the codebase (connect.go, disconnect.go, run_hooks.go) for the double-send-on-channel pattern that caused the original deadlock: none have it.

Testing

• New pkg/phase/manager_test.go: no-deadline/within-deadline completes normally, exceeded-deadline names the hung phase and prevents the next phase from starting, phase errors still propagate with a deadline set.
• make unit-test, make lint, go vet all clean.
• Live verification Configure WhiteSource for GitHub.com #1 (truncated normal operation): provisioned a single-manager stack via examples/terraform/aws-simple, ran launchpad apply --timeout 20s. Ran through 6 real phases, failed cleanly 20.2s in while Prepare hosts (which would have finished on its own) was running:

  ERRO phase 'Prepare hosts' exceeded the overall deadline of 20s
  FATA failed to apply cluster: failed to apply MKE: exceeded overall deadline of 20s while running phase "Prepare hosts"
  

• Live verification Move install command to cmd package #2 (genuine unbounded hang): pointed the host address at 192.0.2.1 (RFC 5737 TEST-NET-1, guaranteed unroutable) and ran launchpad apply --timeout 30s. The underlying SSH dial's own i/o timeout took real, non-trivial time per attempt (2 failed attempts logged within the 30s window); the deadline correctly cut off Open Remote Connection at 30.2s — well before the phase's own 60-attempt retry budget would have exhausted naturally, which at this per-attempt duration could run considerably longer than expected:

  ERRO 192.0.2.1:22: attempt 1 of 60.. failed to connect: ... ssh dial: dial tcp 192.0.2.1:22: i/o timeout
  ERRO 192.0.2.1:22: attempt 2 of 60.. failed to connect: ... ssh dial: dial tcp 192.0.2.1:22: i/o timeout
  ERRO phase 'Open Remote Connection' exceeded the overall deadline of 30s
  FATA failed to apply cluster: failed to apply MKE: exceeded overall deadline of 30s while running phase "Open Remote Connection"
  

• Live verification Dev environment via footloose #3 (no false positive on the happy path): full, unmodified launchpad apply at the default 90m timeout against a real single-manager stack. Completed successfully — all 42 MKE install steps, "Cluster is now configured" — in 3m59.682s, far under the deadline. Confirms the mechanism does not interfere with normal operation.
• All AWS infrastructure from every run torn down afterward (terraform destroy, 28/28 resources each time, confirmed no orphans via terraform state list and a tag-filtered describe-vpcs).

Links

• JIRA: PRODENG-3633

Checklist

• Tests added or updated
• Docs updated if user-visible behaviour changed
• No debug output or dead code left in

[james-nesbitt]

Added two more live AWS verifications beyond the original truncated-phase run, since that one only proved the deadline could cut off a phase that would have finished on its own — not that it catches a real unbounded hang, and not that it stays out of the way on a legitimate successful apply.

Genuine hang, not just a short timeout on normal work: pointed the host at 192.0.2.1 (RFC 5737 TEST-NET-1, guaranteed unroutable). The underlying SSH dial's own i/o timeout took real, non-trivial time per attempt — an actual stall, not an artificially truncated fast phase. --timeout 30s correctly cut off Open Remote Connection at 30.2s, well before the phase's 60-attempt retry budget would have exhausted on its own.

No false positive on the happy path: full, unmodified launchpad apply at the default 90m timeout against real infrastructure. All 42 MKE install steps completed, cluster fully configured, in 3m59s — confirms the mechanism doesn't interfere with normal operation.

All AWS infrastructure from both runs torn down and confirmed clean afterward. Full details and log excerpts in the PR description's Testing section.

Written by AI: claude-sonnet-5