feat(workflows): add loop.command for loading loop prompts from files (#1759)

[marc0der]

Summary

• Problem: A workflow loop node could only carry its iteration prompt inline (loop.prompt), with no way to externalise it to a command file — even though every non-loop AI prompt has this escape hatch via command: nodes (loadCommandPrompt).
• Why it matters: The longest prompts in workflows (Ralph-style implement/build loops, iterate-until-valid loops) are exactly the ones stuck inline. The bundled archon-ralph-dag.yaml inlines ~460 lines in its implement loop for this reason.
• What changed: Added an optional loop.command field that loads the iteration prompt from a command file via the existing loadCommandPrompt resolver. Mutually exclusive with loop.prompt — exactly one is required. Schema, executor, validator, web types, canvas label, docs, and tests at every layer.
• What did NOT change (scope boundary): No bundled workflows refactored. No generic prompt_file: for other node types. No visual loop-node editor (none exists today — builder involvement is limited to API types + canvas label). No DB changes, no new external calls.

UX Journey

Before

Workflow author wants to externalise a long loop prompt:
  author ──▶ loop.prompt: |  (~460 lines inline in YAML)
  author ◀── no alternative; stuck inline

After

Workflow author wants to externalise a long loop prompt:
  author ──▶ extract prompt to .archon/commands/my-loop.md
  author ──▶ loop.command: my-loop                                [+]
  engine  ──▶ loadCommandPrompt() (repo → home → bundled)
  engine  ──▶ reuses loaded text every iteration, with variable substitution

Architecture Diagram

Before

loop node    ──▶ loop.prompt (inline string)
              ──▶ substituteWorkflowVariables() ──▶ AI iteration

command node ──▶ loadCommandPrompt() ──▶ command file body ──▶ AI step
                  (repo → home → bundled)

After

loop node    ──▶ loop.prompt (inline string) ─────────────────┐
              ──▶ loop.command [+] ──▶ loadCommandPrompt() ───┤
                                       (read once at node start) ──▶ substituteWorkflowVariables()
                                                                      ──▶ AI iteration (reused per iteration)

command node ──▶ loadCommandPrompt() (unchanged)

Connection inventory:

From	To	Status	Notes
executeLoopNode (dag-executor.ts)	loadCommandPrompt (executor-shared.ts)	new	Same resolver command: nodes use; read once before iteration loop
loopNodeConfigSchema (schemas/loop.ts)	isValidCommandName (command-validation.ts)	new	Defense-in-depth path-traversal check at parse time
validateWorkflowResources (validator.ts)	resolveCommand + findSimilar	new	Mirrors the command-node check; same hints + "did you mean"
validateDagStructure (loader.ts)	loop.prompt only	modified	$nodeId.output ref scan skips command-loaded text (parse-time can't read it)
resolveNodeDisplay (web/dag-layout.ts)	dn.loop.command	new	Labels command-backed loops by command name
api.generated.d.ts	loop.{prompt?, command?}	modified	Regenerated from the schema; exactly-one enforced server-side

Label Snapshot

• Risk: risk: low
• Size: size: M
• Scope: workflows|web|docs|tests
• Module: workflows:loop

Change Metadata

• Change type: feature
• Primary scope: workflows

Linked Issue

• Closes feat(workflows): let loop nodes load their prompt from a command file (loop.command:) #1759

Validation Evidence (required)

bun run validate
# All gates green:
# ✅ check:bundled       — 36 commands, 20 workflows up to date
# ✅ check:bundled-skill — 21 files up to date
# ✅ type-check          — all 10 packages clean
# ✅ lint                — 0 errors, 0 warnings (--max-warnings 0)
# ✅ format:check        — all files formatted
# ✅ test                — 0 fail across every package
#                          @archon/workflows: 614 tests
#                          (+5 dag-executor runtime cases, +4 loader cases, +5 validator cases, +1 web canvas case)

• Evidence provided: full bun run validate exit 0 locally; live end-to-end test documented under Human Verification.
• No commands intentionally skipped.

Security Impact (required)

• New permissions/capabilities? No
• New external network calls? No
• Secrets/tokens handling changed? No
• File system access scope changed? No — loop.command resolves through the same loadCommandPrompt resolver command: nodes already use (repo → home → bundled), with the same isValidCommandName path-traversal validation. Nothing new is reachable.

Compatibility / Migration

• Backward compatible? Yes — purely additive; every existing loop.prompt keeps working unchanged.
• Config/env changes? No
• Database migration needed? No
• Upgrade steps: none.

Human Verification (required)

What was personally validated beyond CI:

• Verified scenarios: End-to-end live test on a real workflow in a separate repo. The ralph-wiggum Archon workflow (an autonomous plan/build loop) has two loop: nodes whose inline prompts run ~55 and ~75 lines. I extracted those bodies into .archon/commands/ralph-plan.md and ralph-build.md, rewired the workflow to use loop.command: ralph-plan / loop.command: ralph-build, hard-reset a throwaway branch to immediately after the spec commit / before the implementation of a real feature (semverish-version-validation), then ran the converted workflow against that state from a local source build of this branch. Both loops loaded their command files, drove real iterations with full variable substitution ($LOOP_PREV_OUTPUT, $LOOP_USER_INPUT, $ARGUMENTS, $nodeId.output), and produced the expected per-iteration behaviour. No loop.command failures, parse errors, or substitution issues observed.
• Edge cases checked: Both-prompt-and-command and neither-defined reject at parse time with field-targeted errors (loader tests). Unsafe command names (e.g. ../escape) reject at both parse time and validate time. Missing / empty / unreadable command targets fail the node fast with actionable messages mirroring the command-node failure shape. Read-once invariant proven behaviourally by deleting the source file mid-iteration and confirming subsequent iterations still complete.
• What was not verified: A binary-build smoke run — the change is engine-layer; the binary build path was not exercised manually, but check:bundled and the embedded-defaults regeneration pipeline are covered by CI.

Side Effects / Blast Radius (required)

• Affected subsystems/workflows: @archon/workflows (schema, loader, validator, executor) and @archon/web (regenerated API types + canvas label). No backend route, DB, or other-package surface.
• Potential unintended effects: None observed. The new path mirrors the existing command: node path so closely that any regression would surface in the existing command-node tests too.
• Guardrails/monitoring: Schema rejects both/neither at parse time; validator rejects missing-file before a run; executor fails fast with node_failed on any runtime resolution failure (same observability shape as a missing command: node file).

Rollback Plan (required)

• Fast rollback command/path: Revert this PR (or the merge commit). No DB state, no migrations, no on-disk artefacts.
• Feature flags or config toggles: None needed — loop.command is opt-in per node. Removing it from a workflow falls back to loop.prompt semantics.
• Observable failure symptoms: Workflows using loop.command would fail at validation time ('<name>' command not found) or with a clear node_failed event at runtime.

Risks and Mitigations

• Risk: A future refactor of loadCommandPrompt could change resolution precedence and inadvertently affect command-backed loops differently from command nodes.
  • Mitigation: Both call paths share the same resolver — there's no separate code path to drift. The validator and executor tests for loop.command mirror the command-node tests at every layer.
• Risk: Workflow authors might miss that a missing command file fails the node rather than warning, in cases where they expected silent fallback.
  • Mitigation: Matches existing command: node behaviour exactly. Documentation in loop-nodes.md calls out the fail-fast semantics. Validator surfaces the missing file with "did you mean…" suggestions before the workflow ever runs.

Summary by CodeRabbit

• New Features
  • Loop nodes can now run iterations using a command file (loop.command) instead of an inline prompt (loop.prompt); the UI identifies command-backed loops by the command name and uses the same variable substitution as inline prompts.
• Bug Fixes
  • Workflows now require exactly one of loop.prompt or loop.command, reject invalid/missing/unsafe command targets at load time, and load command text once per node (mid-run file edits are ignored).
• Documentation
  • The Loop Nodes guide was updated with loop.command rules, precedence, and examples.

[coderabbitai]

📝 Walkthrough

Walkthrough

Adds support for loading loop iteration prompts from command files via loop.command, enforces exactly-one-of loop.prompt/loop.command in the schema, preloads command text once at node start for iterations with variable substitution, validates/resolves commands at load time, and updates types, UI, tests, and docs.

Changes

Loop Node Command File Feature

Layer / File(s)	Summary
Loop Node Schema Contract and Type Updates packages/workflows/src/schemas/loop.ts, packages/web/src/lib/api.generated.d.ts	loopNodeConfigSchema adds optional command and makes prompt optional; superRefine enforces exactly-one-of and validates command names. OpenAPI/TS types updated to reflect prompt?: string and command?: string.
Executor: Load and Iterate with Resolved Prompt packages/workflows/src/dag-executor.ts	executeLoopNode now accepts workflow-level command options, pre-loads loop.command via loadCommandPrompt once at node start (fail-fast with node_failed on load error), and applies per-iteration substitutions to the resolved template. Caller updated to forward new args.
Loader and Validator: Reference Scanning and Resource Checks packages/workflows/src/loader.ts, packages/workflows/src/validator.ts	Loader only scans $nodeId.output in inline loop.prompt; validator adds Level 3 checks for loop.command name validity and resolution (repo→home→bundled), emitting errors with hints/suggestions when unresolved or unsafe.
Executor Runtime Tests packages/workflows/src/dag-executor.test.ts	Tests cover read-once command-file loading and reuse across iterations (including mid-run deletion), fail-fast behavior for missing/empty/unsafe commands (no sendQuery), and correct application of loop variable substitutions for command-backed prompts.
Validator Tests packages/workflows/src/validator.test.ts	Resource validation tests for loop.command: repo-local resolution success, missing-command error reporting with suggestions, unsafe-command-name rejection, and home-scoped command resolution with environment setup/teardown.
UI Display Layer packages/web/src/lib/dag-layout.ts, packages/web/src/lib/dag-layout.test.ts, packages/web/src/experiments/console/builder/variants/loop.ts	resolveNodeDisplay shows the loop.command value as the node label for command-backed loops, preserving nodeType: 'loop'; builder deserialization uses nullish fallback for optional prompt; tests added.
Loop Nodes User Guide Documentation packages/docs-web/src/content/docs/guides/loop-nodes.md	Guide updated to document loop.command, resolution precedence, safety rules, read-once semantics, failure behaviors, substitution parity with inline prompts, and usage examples.

Sequence Diagram

sequenceDiagram
  participant Loader
  participant Validator
  participant Executor
  participant loadCommandPrompt
  participant AI
  Loader->>Loader: Parse workflow YAML
  Loader->>Validator: Validate schema (exactly one of prompt/command)
  Validator->>Validator: Resolve/validate loop.command resource
  Executor->>loadCommandPrompt: Load command file once at node start
  loadCommandPrompt-->>Executor: Return command text template
  loop For each iteration
    Executor->>Executor: Substitute variables into template
    Executor->>AI: Send substituted prompt
  end

Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related PRs

• coleam00/Archon#1744: Overlaps with this PR's resolveNodeDisplay changes for loop node labeling in the web DAG builder.
• coleam00/Archon#1367: Related to loop-variable substitution ($LOOP_PREV_OUTPUT) behavior used by command-backed loop prompts.

Poem

🐰 I fetched a prompt from .archon/ land,
I read it once and held it in my hand.
Whether inline or named, the rabbit sings,
Substitutes the vars and flaps its wings.
One of two choices—now the loop hops grand. 🎉

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 75.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and concisely summarizes the main feature: adding loop.command to load loop prompts from files, with the issue number for reference.
Description check	✅ Passed	The PR description comprehensively covers all template sections: problem, rationale, changes, scope boundaries, UX journey, architecture diagrams, validation evidence, security impact, compatibility, human verification, side effects, and rollback plan. All required sections are detailed and substantive.
Linked Issues check	✅ Passed	The PR fully satisfies issue #1759's definition of done: adds loop.command with mutual exclusivity to loop.prompt, enforces exactly-one validation, reuses loadCommandPrompt resolver with identical variable substitution, reads command once at node start, validates command names against path traversal, surfaces missing-file errors with suggestions, and provides comprehensive test coverage.
Out of Scope Changes check	✅ Passed	All changes are directly aligned with the stated PR objectives. Schema, executor, validator, loader, and web changes all relate to implementing loop.command. Documentation and tests support the feature. No bundled workflows were refactored; no generic prompt_file mechanism was added; no database or unrelated modules were modified.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@packages/workflows/src/dag-executor.ts`:
- Around line 1791-1836: The executeLoopNode function currently can emit a
node_failed event on command-load failure without first emitting node_started
and omits the command name in the workflow event payload; fix by emitting a
node_started workflow event (matching other nodes' shape: type: 'node_started',
runId: workflowRun.id, nodeId: node.id, nodeName: node.id) immediately at the
start of executeLoopNode (before resolving loop.prompt/loop.command), and when
handling promptResult.failure (the branch that logs
'loop_node.command_load_failed'), include the failing command string
(loop.command) in the createWorkflowEvent data alongside the error (e.g., {
error: errMsg, command: loop.command }) and also include it in the emitted
getWorkflowEventEmitter() payload so structured logs and events carry the same
context.

In `@packages/workflows/src/schemas/loop.ts`:
- Around line 49-53: The schema currently trims only for validation but leaves
the stored value untrimmed, causing later resolution to fail; update the
loop.command schema to normalize the input (trim and possibly collapse
whitespace) before validation by using a z.preprocess or z.string().transform
that returns (val as string).trim(), then run isValidCommandName against that
normalized value (referencing the hasCommand check and isValidCommandName usage)
and ensure ctx.addIssue message and path reflect the trimmed value; this will
store the normalized command in the parsed output so downstream
resolution/execution sees the trimmed name.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 2f039a14-0167-4790-9413-e40fd652b6e4

📥 Commits

Reviewing files that changed from the base of the PR and between 50beeec and beebceb.

📒 Files selected for processing (11)

• packages/docs-web/src/content/docs/guides/loop-nodes.md
• packages/web/src/lib/api.generated.d.ts
• packages/web/src/lib/dag-layout.test.ts
• packages/web/src/lib/dag-layout.ts
• packages/workflows/src/dag-executor.test.ts
• packages/workflows/src/dag-executor.ts
• packages/workflows/src/loader.test.ts
• packages/workflows/src/loader.ts
• packages/workflows/src/schemas/loop.ts
• packages/workflows/src/validator.test.ts
• packages/workflows/src/validator.ts

[coderabbitai]

Actionable comments posted: 0

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@packages/web/src/experiments/console/builder/variants/loop.ts`:
- Around line 22-26: The loop conversion functions loopFromDag and loopToDag
create a lossy round-trip for command-backed loops. In loopFromDag, preserve the
command field from the input instead of collapsing all command-backed loops to
an empty prompt string. In loopToDag, serialize the command field alongside or
instead of prompt based on which one is present. Add an exactly-one-of
constraint to the builder's loop model to enforce that either prompt or command
is specified, but not both and not neither, matching the schema contract used in
workflows.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 85f92223-3a34-4562-baa1-6736b2137f5f

📥 Commits

Reviewing files that changed from the base of the PR and between f673a61 and ac6f103.

📒 Files selected for processing (12)

• packages/docs-web/src/content/docs/guides/loop-nodes.md
• packages/web/src/experiments/console/builder/variants/loop.ts
• packages/web/src/lib/api.generated.d.ts
• packages/web/src/lib/dag-layout.test.ts
• packages/web/src/lib/dag-layout.ts
• packages/workflows/src/dag-executor.test.ts
• packages/workflows/src/dag-executor.ts
• packages/workflows/src/loader.test.ts
• packages/workflows/src/loader.ts
• packages/workflows/src/schemas/loop.ts
• packages/workflows/src/validator.test.ts
• packages/workflows/src/validator.ts

💤 Files with no reviewable changes (10)

• packages/web/src/lib/api.generated.d.ts
• packages/web/src/lib/dag-layout.test.ts
• packages/web/src/lib/dag-layout.ts
• packages/workflows/src/loader.test.ts
• packages/workflows/src/validator.ts
• packages/workflows/src/loader.ts
• packages/workflows/src/validator.test.ts
• packages/workflows/src/dag-executor.ts
• packages/workflows/src/dag-executor.test.ts
• packages/workflows/src/schemas/loop.ts

✅ Files skipped from review due to trivial changes (1)

• packages/docs-web/src/content/docs/guides/loop-nodes.md

[coderabbitai]

⚠️ Potential issue | 🟠 Major | 🏗️ Heavy lift

Command-backed loops are lossy in builder round-trips

loopFromDag (Line 25) collapses command-backed loops to prompt: '', and loopToDag (Line 41) always serializes prompt while never re-emitting loop.command. This can silently rewrite a valid loop.command node into a prompt-based node on save.

Suggested direction

- export interface LoopNodeData {
-   prompt: string;
+ export interface LoopNodeData {
+   prompt?: string;
+   command?: string;
    until: string;
    max_iterations: number;
    fresh_context: boolean;
    until_bash?: string;
    interactive?: boolean;
    gate_message?: string;
  }

  return {
-   prompt: loop.prompt ?? '',
+   prompt: loop.prompt,
+   command: loop.command,
    until: loop.until,
    max_iterations: loop.max_iterations,
    fresh_context: loop.fresh_context,
    ...
  };

  loop: {
-   prompt: data.prompt,
+   ...(data.command ? { command: data.command } : { prompt: data.prompt ?? '' }),
    until: data.until,
    max_iterations: data.max_iterations,
    fresh_context: data.fresh_context,
    ...
  }

Also enforce exactly-one-of in the builder model (same schema contract as workflows).

Also applies to: 38-42

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/web/src/experiments/console/builder/variants/loop.ts` around lines
22 - 26, The loop conversion functions loopFromDag and loopToDag create a lossy
round-trip for command-backed loops. In loopFromDag, preserve the command field
from the input instead of collapsing all command-backed loops to an empty prompt
string. In loopToDag, serialize the command field alongside or instead of prompt
based on which one is present. Add an exactly-one-of constraint to the builder's
loop model to enforce that either prompt or command is specified, but not both
and not neither, matching the schema contract used in workflows.