Add an optional frozen-harness autoresearch loop for measurable local optimizations

[kvnloo]

Summary

fx has evals (tests/evals/) and skills, but no built-in experiment loop that repeatedly edits a workspace against a frozen local benchmark, keeps or discards the change, and resumes after restart.

This is a feature request for an optional, CLI-first autoresearch mode. It is not a request to turn fx into an IDE or to add a dashboard.

Why this fits fx

fx is already used as a cheap, many-instance worker (fx ask --json --no-save). The missing piece is a durable keep/discard loop for work that has a mechanical score (startup time, binary size, test latency, token count on a frozen prompt). That is the same class of leftover cost fx itself targets: measure, change one thing, keep only if the number improves.

This should stay optional and small. Interactive fx remains the default.

Prior art (not a copy-paste spec)

Oh My Pi (OMP) implements this as a core extension, not only a skill:

• Slash command: /autoresearch [goal], /autoresearch off, /autoresearch clear [--keep-tree|--reset-tree]
• Dedicated autoresearch/* git branch
• SQLite session/run state under ~/.omp/autoresearch/
• Frozen project harness, conventionally ./autoresearch.sh
• Tools: init_experiment, run_experiment, log_experiment, update_notes
• Parse METRIC name=value and ASI key=value from harness output
• Lifecycle: baseline, KEEP, DISCARD, CRASH, CHECKS_FAILED
• Commit kept deltas; revert only the rejected experiment
• Auto-continue until interrupt or budget

Hermes has a local, unmerged port of that contract. One lesson from that port: do not dynamically swap the model tool schema mid-session the way OMP uses setActiveTools. fx should keep a stable tool/prompt surface per session.

I am not proposing that fx import OMP or Hermes code. I am proposing the same job, shaped to fx.

Proposed fx shape (for discussion)

Prefer the smallest owner that matches current architecture:

1. v0 skill + fx ask: a managed skill that tells the agent to run a frozen ./autoresearch.sh, write KEEP/DISCARD notes, and stay on an autoresearch/* branch. Good for proving demand. Weak on resume/crash recovery.
2. v1 builtin slash command (recommended if this lands in tree): /autoresearch plus durable state under ~/.fx/autoresearch/<workspace-key>.db, using existing session/permission/git tools. fx ask --json should be able to drive one iteration unattended.
3. Not proposed: dashboard UI, dynamic tool-set mutation, unattended writes outside a worktree, or replacing tests/evals/.

Hard constraints I think fx should keep:

• Frozen evaluator identity (the harness file is not part of the experiment diff)
• Fail closed on dirty/ambiguous VCS
• Scope/off-limits fail closed
• Text and JSON output for every control command
• Permission-first: --auto may run the harness, but repo writes stay on the isolated branch
• No live-model evals as the gate (existing Full CI rule)

Best first metric for fx itself: zig build wall time, zig-out/bin/fx size, or fx ask --no-save cold start on a fixture prompt. Those are local, deterministic, and on-mission.

What I already checked

• No open issue or source hit for autoresearch in vercel-labs/fx
• CONTRIBUTING: CLI-first, small reviewable changes, typed contracts, draft PR until Full CI is green on the exact commit
• Skills can be installed to ~/.fx/skills/, but they are instructions, not a crash-safe experiment ledger
• kvnloo/fx is a current fork of this repo; I can implement a slice after design agreement

Ask

Is a skill-only experiment acceptable as v0, or do you want a builtin /autoresearch owned by src/core/ if the contract is small enough?

I will not open an implementation PR until a maintainer says which owner module and persistence contract you want.