GSO optimizer-v2: engine rewrite + Genie Agent rename - #315
Open
hiydavid wants to merge 198 commits into
Open
Conversation
…Phase 2 §3.6) Add an always-on, deterministic exclusion that prevents ANY scored benchmark Q/A from being seeded into a Genie Space's Example SQL Queries section — including examples derived from passing benchmark rows. LeakageOracle.is_scored_benchmark_qa matches a candidate by question-id, normalized-SQL hash (canonicalize_sql), OR canonical question text (new canonical_question_key). Because the corpus is the whole scored set (no train/held-out split, D8), passing rows are covered identically to failing ones. is_example_sql_benchmark_leak (the applier-side proactive firewall used by _apply_proactive_example_sqls) now hard-blocks on this deterministic match BEFORE, and independent of, the tunable fuzzy evaluate_example_sql policy and GSO_EXAMPLE_SQL_FIREWALL_STRICT — closing the relaxed-mode path where a verbatim benchmark Q/A could otherwise be applied as a teaching example. evaluate_example_sql's existing fuzzy contract is unchanged. Co-authored-by: omnigent <noreply@omnigent.ai>
…se 2) Extract the merge-only publish core into publish_benchmarks_to_genie_space_with_report, returning a BenchmarkPushReport (added rows with id/question/sql, dedup/mirror skips, truncation, merged total). The preflight push and the §3.5 provenance ledger consume this structured report instead of re-deriving the diff. publish_benchmarks_to_genie_space is now a thin wrapper returning report.added_count — every existing caller and test is unchanged. Merge semantics (additive, user-authored rows preserved, example-SQL mirror exclusion) are identical. Co-authored-by: omnigent <noreply@omnigent.ai>
…§3.5)
Add genie_opt_benchmark_mutations(run_id, question_id, op in
{added,removed,changed}, before, after, reason, logged_at): the Delta
provenance ledger of every benchmark mutation GSO makes to the user's live
Genie Space. DDL lives in ddl.py alongside the other genie_opt_* tables and
is registered in _ALL_DDL (so ensure_optimization_tables + grant_permissions
pick it up automatically); CDF-versioned + partitioned by run_id like its
siblings.
state.write_benchmark_mutations is the best-effort writer (validates op,
JSON-serializes before/after, never aborts preflight on failure). Also adds
the BENCHMARK_WINDOW_MIN/MAX (30/40, D8) config constants. The backend
endpoint + UI 'Benchmark changes' view that consume this ledger are Phase 6
(out of scope here).
Co-authored-by: omnigent <noreply@omnigent.ai>
…ase 2 D8) Wire the benchmark push at preflight — runner-independent, before baseline eval (no dependency on the unbuilt Phase-1 EvalRunner seam). New preflight_push_benchmarks_to_space runs as run_preflight.py Step 1d.2: - Push the WHOLE EXPLAIN-validated set additive/merge-only into serialized_space.benchmarks.questions (no train/held-out split). User- authored rows are never deleted. - Prune-invalid backstop before publish: any row not EXPLAIN-valid or lacking ground-truth SQL is dropped, so a SQL-erroring question can never be published. - 30-40 window as a RECOMMENDATION via compute_benchmark_window_recommendation (>40 => near-duplicate-first recommended prune; <30 => synthesis top-up count), surfaced through a PREFLIGHT_BENCHMARK_WINDOW stage. Never a silent auto-delete; the existing <TOP_UP_THRESHOLD synthesis still tops up. - Populate the genie_opt_benchmark_mutations ledger (added from the push report, removed from validation/backstop prunes, changed from predicate auto-corrections). preflight_validate_benchmarks now also returns the rejected/changed sets that feed the ledger. The preflight config_snapshot taken at run start remains the discard revert anchor — rollback semantics are unchanged. The push is best-effort: a failure is non-fatal and baseline can still score the space's existing set. Co-authored-by: omnigent <noreply@omnigent.ai>
Introduce the official Databricks Genie Benchmark (Eval-Run) API as the active eval path (decision D1), replacing the in-process scorer path so we never double-run. Engine-only Python; frontend untouched. - optimization/eval_runner.py: EvalRunner protocol + OfficialBenchmarkRunner (create -> poll -> list(paginated) -> details, injectable clock/sleep). map_eval_detail_to_row maps each official result into the EXISTING flat per-question row dict (no parallel schema); accuracy = num_correct/ num_questions; verdict from `assessment`; reasons from `assessment_reasons`. build_eval_output_from_official produces the legacy run_evaluation contract; resolve_space_benchmark_qids best-effort maps GSO benchmarks -> space qids. - optimization/eval_budget.py: EvalBudget cumulative-wall-clock guard that reserves the finalize run; estimate_three_gate_seconds; assess_working_set (30-40 recommendation). Eval-runs are sequential -> budget = sum of runs. - optimization/eval_gates.py: run_three_gate subset-first slice->P0->full sequencing with short-circuit + per-gate budget recording. - evaluation.run_evaluation: route through the official runner behind the USE_OFFICIAL_BENCHMARK_RUNNER switch (default ON, real WorkspaceClient only), returning before mlflow.genai.evaluate() so the 9 in-process judges never run alongside it. Falls back to legacy when the subset can't be resolved or on any official-path error. - harness._run_lever_loop: instantiate the budget; record each iteration's eval wall-clock; stop before an iteration when the remaining wall (after the finalize reserve) can't fund another gate cycle. max_iterations is an upper bound, not a target. Deferred to Phase 3: retire the 9 judges, reason->lever repoint, acceptance/ threshold rework. Deferred to Phase 2: guaranteed benchmark push + robust qid resolution, example-SQL leakage guard, provenance ledger. Gates: pytest +39 new unit tests (3862 passed; the 4 failures are pre-existing on the base branch); `uv run ty check` adds 0 new diagnostics. Co-authored-by: Isaac
Both fail on gso/optimizer-v2 independent of Phase 2; repaired so the gate is green: - test_state_migration::test_rolled_back_entry_is_present_in_real_migrations introspected _migrate_add_columns source, but the migration list was refactored into ddl.ADDITIVE_COLUMN_MIGRATIONS (the loop iterates it). The test now inspects the real list — preserving (and strengthening) the guard that the rolled_back BOOLEAN DEFAULT false entry is present. - test_skill_parser_handoff hard-failed with FileNotFoundError when the untracked docs/skills/ fixtures are absent. Per the documented integration-test contract (self-skip without fixtures), the doc reader now pytest.skip()s when a SKILL.md is missing; the parser-API smoke test still runs and the assertions stay live wherever the fixtures exist. Co-authored-by: omnigent <noreply@omnigent.ai>
Cross-cutting (D1): once genie_create_eval_run is called the official path fails closed — never fall back to legacy mlflow.genai.evaluate(), never treat a partial/failed/empty result as a pass. Fallback is allowed ONLY before any eval-run is created. F1 — subset-first 3-gate is now ACTIVE in Phase 1 (was skip-to-full). When the official runner is the eval path, harness._run_gate_checks runs slice -> P0 -> full using the capped §3.4 selectors (eval_gates.select_slice_qids / select_p0_qids; caps SLICE_GATE_MAX_QUESTIONS=10 / P0_GATE_MAX_QUESTIONS=15); full runs ONLY after slice+P0 pass (each short-circuits to rollback on regression). Existing acceptance thresholds unchanged — only the sequencing is added (threshold rework stays Phase 3). Mocked-workspace tests keep the legacy path (runner is None). F2 — fail-closed after creation (evaluation.run_evaluation): split into Phase A (pre-creation: build runner + resolve qids; any failure -> legacy fallback, no eval-run created) and Phase B (post-creation: run() + return, NOT wrapped in a fallback try/except). No double-run. F3 — complete qid resolution required (resolve_space_benchmark_qids): returns the full list only when EVERY requested benchmark resolves; any unresolved -> None -> legacy fallback before creation. No silent partial runs. F4 — non-DONE/partial/empty never reads green (EvalRunResult.is_complete_success + build_eval_output_from_official): such runs map to accuracy 0, every requested id as a failure, thresholds_met=False -> all gates reject -> rollback. F5 — row-schema legacy aliases (map_eval_detail_to_row): mapped rows now carry inputs/question, outputs/response, inputs/expected_response, generated_sql, expected_sql (the keys active harness/state/feature-mining readers consume), plus the native assessment/assessment_reasons. Added a row-schema compat test exercising the real harness readers + row_is_hard_failure. Non-blocking: budget recording scoped to the official eval-run wall-clock (eval_runner accumulator reset/summed per iteration) instead of whole-gate elapsed; TODO wording corrected (working-set window enforcement is Phase 2). Gates: pytest 3874 passed (+12 new tests; same 4 pre-existing base-branch failures); uv run ty check 392 diagnostics, 0 new. Co-authored-by: Isaac
Mark all five Phase 2 checkboxes complete with implementation pointers and append a dated §6 progress-log entry describing the benchmark-lifecycle push, 30-40 window recommendation, example-SQL leakage guard, and provenance ledger shipped in this PR. Co-authored-by: omnigent <noreply@omnigent.ai>
Round-2 cross-review caught that EvalRunResult.is_complete_success did not enforce non-empty collected rows: a nominally-DONE run with num_questions>0 but rows==[] (a pagination/listing quirk, or every detail failing to map) read as complete success, so build_eval_output_from_official used the server-reported accuracy with an EMPTY failure set and the gate passed green — re-opening the round-1 F4 hole through a different trigger. Fix: add `len(self.rows) >= self.num_questions` to is_complete_success. The official path collects exactly one row per benchmark question (1:1 results->rows; a per-detail mapping failure raises rather than silently shortening), so a genuine success always satisfies the check; short/empty row sets now fail closed exactly like a non-DONE status (accuracy 0, all requested ids -> failures, thresholds_met=False -> gate rejects -> rollback). Tests: +2 (test_done_with_empty_rows_fails_closed, test_done_with_short_rows_fails_closed) next to test_empty_run_fails_closed; updated test_complete_done_is_success to supply rows. Gates: pytest 3876 passed (same 4 pre-existing base-branch failures, 0 new); uv run ty check 392 diagnostics, 0 new. Co-authored-by: Isaac
…uard, fatal push (Phase 2 review) Address 4 BLOCKING findings from the cross-vendor (Codex) review of PR #234 against the Phase 2 acceptance contract. BLOCKING 1 — live-space truncation removed. The benchmark publisher (publish_benchmarks_to_genie_space_with_report) is now strictly additive/merge-only: it never slices the merged set, so pushed rows can't be silently dropped and existing user-authored rows can't be deleted. `max_questions` is the genuine Genie API hard cap (GENIE_MAX_BENCHMARK_QUESTIONS=500), not the train/held-out target; if the merged set would exceed it the publisher FAILS CLOSED (over_cap report, no patch) instead of truncating. preflight/run_preflight no longer pass the train/held-out `effective_max` as the live publisher cap. BLOCKING 2 — the 30–40 window recommendation is now computed over the POST-MERGE set (existing live rows + net-new additions, after dedupe) returned by the publisher, before patching — not over the pre-merge `pushable` slice. A 25-existing + 30-new space now correctly reports over_window on 55. BLOCKING 3 — the deterministic scored-benchmark Q/A hard-block (question-id / canonical question text / normalized-SQL hash) is centralized in leakage.deterministic_scored_benchmark_qa_leak and now covers EVERY example-SQL write path: folded into is_benchmark_leak (Lever-5 _validate_lever5_proposals + synthesis validate_synthesis_proposal), reused by is_example_sql_benchmark_leak (proactive seeding), and applied last-mile in applier.apply_patch_set (the normal apply path; wired at the lever-loop and proactive apply sites). BLOCKING 4 — a required preflight push failure (publisher raised, or merged set over the hard cap) now raises BenchmarkPushError and fails the preflight job, so baseline eval can never run against the stale live benchmark set. run_preflight re-raises instead of swallowing. Also: the provenance ledger records over-window prune RECOMMENDATIONS (op=prune_recommended, advisory/non-mutating) plus the net-new push set; stdout truncates the prune list to 20 ids while the full list lands in the stage detail and ledger. New tests exercise the REAL merge-only publisher (preserve existing, push 31-40 without truncation, over-window recommend-only, post-merge overflow no-deletion, mirror-skip, hard-cap fail-closed) and the deterministic guard on the Lever-5, synthesis, and applier write paths. Co-authored-by: Isaac
…ase1 GSO v2 — Phase 1: EvalRunner swap to native Benchmark API
…-phase2-benchmark-lifecycle
…nchmark-lifecycle GSO Optimizer v2 — Phase 2: Benchmark-question lifecycle into the live space
…ting, API-accuracy gating (D2)
Retire the 9 scored LLM judges from the GSO Optimizer v2 decision path. The
official Databricks Genie Benchmark API verdict is now the sole quality signal;
its assessment_reasons drive lever routing.
- Routing: rca._ASSESSMENT_REASON_TO_RCA_KIND maps all 25 official ScoreReason
values + the derived ASSET_TYPE_MISMATCH to an existing RcaKind; levers come
from the established _RCA_KIND_TO_LEVERS (no new lever map). Handles the 6
previously-unmirrored LLM_JUDGE_* reasons. Wired into
extract_rca_findings_from_row, gated on the official-runner-only
assessment_reasons key so the legacy/mocked path is untouched. The
deterministic SQL-shape RCA stays as the fine sub-router (reason findings at
confidence 0.6 < SQL-shape 0.8-0.9). EMPTY_GOOD_SQL flagged non-actionable.
- Asset-type nugget: eval_runner._asset_type_annotations derives
expected/actual_asset_type + asset_type_mismatch on BAD/NEEDS_REVIEW rows via
detect_asset_type; rca turns a mismatch into a Lever-5 finding. GOOD rows none.
- Acceptance: DEFAULT_THRESHOLDS collapsed to the single API-accuracy gate
{result_correctness: 85.0}; all_thresholds_met re-documented + overall_accuracy
alias; decide_acceptance already pure accuracy-delta (unchanged).
- Judges marked retired (scorers/__init__.py docstring + RETIRED_JUDGES);
physical scorer-module deletion deferred to Phase 7.
Tests: +25 (tests/unit/test_phase3_reason_routing.py + asset-annotation tests in
test_eval_runner.py). Gates: GSO pytest 3934 passed / 17 skipped / 3 xfailed;
uv run ty check 392 diagnostics (all pre-existing, 0 new).
Co-authored-by: Isaac
…ion marker (D3) Engine-only, fully additive (legacy/mocked path unchanged). Closes the one D3 gap: capture the FULL effective Genie Space config per iteration in Delta, and mark the champion iteration in Delta — no UC model registration. - ddl.py: add `config_json` STRING + `is_champion` BOOLEAN columns to genie_opt_iterations (declared in the DDL for fresh installs AND registered in ADDITIVE_COLUMN_MIGRATIONS for existing tables). Chose a column over a new genie_opt_configs table: 1:1 with iteration rows (no join), matches the wide-table convention, reuses the additive-migration machinery, inherits the table's already-enabled CDF for versioned history. - state.py: new `config_snapshot` kwarg on write_iteration → serialized to config_json via a self-contained, MLflow-free, whitelist + cycle-safe `_project_config_for_iteration` (drops optimizer-internal `_*` keys). New `mark_champion_iteration` clear-then-set writer (best-effort). Both columns added to _REQUIRED_ITERATION_COLUMNS. - models.py: promote_best_model now calls mark_champion_iteration REUSING its existing idxmax(overall_accuracy) selection, placed before the MLflow model_id guard so the Delta marker lands even on the Delta-only path. No register_uc_model added (Phase 5 owns UC decommission). - harness.py / run_baseline.py: thread config_snapshot into every iteration write site (baseline, enrichment, lever-loop full + slice/p0, held-out). - Rollback/discard confirmed unchanged (pre_snapshot re-PATCH and genie_opt_runs.config_snapshot revert); new columns never enter those paths. - tests: test_phase4_config_tracking.py (22 tests) covering DDL/migration, projection, write path, champion marking + selection reuse, and the rollback/discard no-regression assertions. Gates: pytest 3965 passed / 17 skipped / 3 xfailed (0 failures, +22 new); ty check 392 diagnostics (0 new, 0 in touched files). Co-authored-by: Isaac Co-authored-by: omnigent <noreply@omnigent.ai>
…h the active clustering/lever path (D2)
Closes two blocking cross-review findings: the ACTIVE clustering + lever-
assignment path still routed official Benchmark rows via the legacy
judge/root-cause maps, so assessment_reasons weren't driving the active mapped
lever or the strategist's recommended_levers.
Blocking issue 1 — active mapped lever:
- cluster_failures now captures each official row's top-level assessment_reasons
(+ derived asset_type_mismatch) onto the failure entry and aggregates them
(frequency-ordered, ASSET_TYPE_MISMATCH folded in) onto the cluster as
cluster["assessment_reasons"] — set only for official rows; legacy clusters
never gain the key.
- _map_to_lever gains an assessment_reasons param and, when present, returns the
reason-derived primary lever (rca.levers_for_assessment_reasons) BEFORE the
legacy judge/root-cause/ASI fallbacks. Non-actionable reasons (e.g.
EMPTY_GOOD_SQL) contribute no lever ⇒ legacy fallback.
- The harness _mapped_lever calls (both sites) and the optimizer natural-lever
sites now pass assessment_reasons=cluster.get("assessment_reasons").
Blocking issue 2 — strategist recommended_levers:
- recommended_levers_for_cluster prefers official reason-derived levers over the
root-cause shape defaults; stamp_recommended_levers_on_clusters precedence is
official reasons > existing explicit recommendation > shape defaults (shape
defaults never overwrite official-derived or upstream-explicit values).
Tests: +9 regressions in test_phase3_reason_routing.py exercising the active
cluster_failures → _map_to_lever path and stamp_recommended_levers_on_clusters
on official-row clusters (incl. a case where legacy and official disagree).
Gates: GSO full pytest 3943 passed / 17 skipped / 3 xfailed; uv run ty check
392 diagnostics (all pre-existing, 0 new; none in the changed files).
Co-authored-by: Isaac
…view (D3/D6/D7) Make GSO tracking/versioning Delta-only end-to-end. Removes the MLflow LoggedModel + UC Model Registry paths, the MLflow Review App labeling session, the MLflow Prompt Registry judge-prompt registration gate, and the now-dead MLflow pointer columns / experiment_name job param. Surviving strategist/benchmark/eval MLflow tracing is intact (self-resolved experiment path); cross-env deploy is out of scope (future = DAB genie_space). - Item 1: gut models.py to Delta-only promote_best_model; drop create_genie_model_version / link_eval_scores_to_model / rollback_to_model + the model_creation_kwargs per-mutation-run carrier. - Item 2: remove register_uc_model / _GenieConfigSnapshot / ENABLE_UC_MODEL_REGISTRATION + the dead cross-env deploy notebooks/launcher. - Item 3: split labeling.py — drop the MLflow Review App fns, keep the Delta flagging (NEEDS_REVIEW); scrub labeling_session_* plumbing. - Item 4: drop register_judge_prompts + STRICT_PROMPT_REGISTRATION + the preflight prompt-registry gate; judge prompts stay as config.py constants. - Item 5: scrub best_model_id/experiment_*/labeling_*/mlflow_run_id/model_id columns, the experiment_name job param, and MLflow ResourceLinks. Gates: GSO pytest 3979 passed / 0 failed; ty 391 (0 new vs 392 base); backend 445 passed (2 pre-existing create_agent failures, unrelated); frontend tsc clean, lint 0 net-new, vitest 44 passed. Co-authored-by: Isaac
…son + atomic champion + PII trim (D3) Addresses the cross-vendor review of PR #237 (1 blocking + 3 non-blocking). BLOCKING — lever-loop iteration rows persisted the WRONG config_json. The slice/P0/full gate writes in `_run_gate_checks` passed `config_snapshot=metadata_snapshot`, but the caller sets metadata_snapshot to the PRE-patch rollback anchor before `apply_patch_set`, which deep-copies its input (never mutates it) and returns the evaluated candidate as `apply_log["post_snapshot"]`. So iters 2..N recorded the pre-patch config and the rejected-candidate configs Phase 4 must version were lost. Fix: derive `_candidate_config_snapshot = apply_log["post_snapshot"]` (guarded; falls back to metadata_snapshot only when no post-apply snapshot exists) once at the top of `_run_gate_checks` and use it at all three gate writes. Rollback semantics unchanged — metadata_snapshot stays the rollback anchor; only the recorded config_json changed. Non-blocking: - Atomic champion marking: `mark_champion_iteration` is now a single run-scoped conditional UPDATE (`SET is_champion = (iteration = <best> [AND eval_scope = <scope>])`) instead of clear-then-set, so a row can never be cleared without the new champion being set in the same statement. - Whitelist comment drift corrected: the iteration config whitelist intentionally DIFFERS from models._SAFE_SPACE_CONFIG_KEYS (extends with benchmarks/config, prefers _parsed_space) — comment now says so. - No ACL/PII in Delta: dropped `permissions`/`owner` from the whitelist so ACL/user-identity data never reaches config_json, even for a raw config passed without _parsed_space. Tests: test_phase4_config_tracking.py now 26 — adds a real-gate-path regression test (slice gate forced to fail; asserts config_snapshot IS post_snapshot), an all-three-sites static guard, the atomic-champion assertions, and the ACL/PII-omission tests. Gates: pytest 3969 passed / 17 skipped / 3 xfailed (0 failures); ty check 392 diagnostics (0 new, per-file diff vs clean tree identical). Co-authored-by: Isaac Co-authored-by: omnigent <noreply@omnigent.ai>
GSO Optimizer v2 — Phase 3: judge re-architecture (D2)
…bbed symbols/columns Addresses 3 BLOCKING dangling-reference bugs from the cross-vendor (codex) review of PR #238 — parallel spots missed in the first pass: - FIX 1 (runtime ImportError): jobs/run_preflight.py still imported + called the removed preflight_probe_prompt_registry (former Step 1e.2). Removed the import/call/cell + cleaned stale notebook prose (judge-prompt/LoggedModel summary, experiment_name widget row, labeling_session_url migration example). - FIX 2: backend/services/gso_lakebase.py load_gso_iterations SELECT still listed mlflow_run_id, model_id (would break on fresh post-Phase-5 tables). - FIX 3: backend/models_db.py GSOIterationRecord ORM mirror still declared mlflow_run_id/model_id fields — removed to match the scrubbed DDL. Added 3 regression guards to test_phase5_decommission.py. Phase-6 response fields + the ASI table left untouched (strict scope fence). Gates: GSO pytest 3982 passed / 0 failed; ty 390 (0 new vs 392 base); backend 445 passed (2 pre-existing create_agent failures, unchanged). Co-authored-by: Isaac
feat(gso): Phase 4 — Delta-only per-iteration config tracking + champion marker (D3)
Since May 15, 2026, deleted Lakebase projects enter a 7-day soft-deleted state that reserves their IDs. ensure_project() saw get_project return NotFound, then swallowed the ALREADY_EXISTS from create_project, so the install failed later with an opaque "Project with name ... not found". - Purge a soft-deleted project holding the name before recreating, with a clear warning that the old data is unrecoverable - Stop blindly swallowing ALREADY_EXISTS: re-verify the project is visible and raise an actionable error if the name is reserved by an inaccessible project - Apply the same fix to scripts/setup_lakebase.py (terminal path) - Bump databricks-sdk 0.102.0 -> 0.117.0 (soft-delete APIs need >=0.106) - Set sys.dont_write_bytecode in notebooks/install.py to silence wsfs __pycache__ errors when importing deploy_lib from a Git folder Co-authored-by: Isaac
…nie Workbench)
Migrate the Workbench off the retired per-judge surface onto the native
Benchmark API verdict (GOOD/BAD/NEEDS_REVIEW + assessment_reasons[]) end-to-end.
Engine (enables the contract — data must be in Delta since synced tables are off):
- genie_opt_iterations gains additive num_needs_review / eval_run_id /
eval_run_status columns (DDL + ADDITIVE_COLUMN_MIGRATIONS + _REQUIRED_ITERATION_COLUMNS);
state.write_iteration persists them from build_eval_output_from_official
(back-compat NULL on legacy rows). Phase-4 positional INSERT assertions updated;
+2 write_iteration persistence tests.
Backend (backend/routers/auto_optimize.py):
- question-results: drop the 9-judge list; derive state from `assessment`;
return assessment + assessment_reasons[] (no judge_verdicts).
- /iterations: add num_done/num_correct/num_needs_review; replace thresholds_met
with api_accuracy_gate_met + eval_gate_status.
- /eval-results (+ /asi-results alias): lightweight {question_id, assessment,
assessment_reasons} from rows_json — no genie_eval_asi_results dependency.
- baseline step-detail: assessment/reason summary + evaluationRunUrl + eval-run status.
- /benchmark-changes: serve the genie_opt_benchmark_mutations ledger
(added/removed/changed + provenance).
Frontend:
- Delete JudgePassRates.tsx + Judges tab; remove "9 evaluation judges" copy.
- lib/assessment.ts adds the 3-valued state (incl. NEEDS_REVIEW) across
QuestionList / QuestionDetail / QuestionJourney.
- StepDetailContent → assessment + reason-count summary + evaluationRunUrl.
- Repoint ScoreSummary / RunDetailView / IterationChart to official counts.
- New BenchmarkChangesPanel + tab; TS types migrated (IterationRow,
GSOQuestionDetail, GSOQuestionResult, GSOBenchmarkChanges/Mutation).
Gates: GSO pytest 3984 passed/0 failed; auto_optimize router 30 passed;
frontend tsc + vite build clean, vitest 44 passed; no new eslint errors.
Co-authored-by: omnigent <noreply@omnigent.ai>
get_project returns soft-deleted projects with delete_time set instead of 404ing; only their sub-resources (branches, roles, databases) return NotFound. Check delete_time on the successful GET so ensure_project purges and recreates reserved names, and require_project rejects soft-deleted projects with an actionable error. Verified live against a workspace with a soft-deleted project: purge + recreate succeeds and the branches endpoint resolves afterwards. Co-authored-by: Isaac
…+ official accuracy denominator Addresses 2 blocking items from the Phase 6 cross-review (PR #239), plus the cheap non-blocking follow-ups. No judge-era surface reintroduced. BLOCKER 1 — judge-era baseline summary text: - _build_step_summary (the human-readable step summary, separate from the already-migrated _build_step_io detail) still emitted "with 9 evaluation judges". Rewrote it assessment-centric: official num_correct/num_questions accuracy + a NEEDS_REVIEW count, no mention of judges. BLOCKER 2 — official accuracy denominator end-to-end (num_correct/num_questions): - Backend /iterations recomputes overall_accuracy = num_correct/num_questions for OFFICIAL rows (detected via eval_run_id/eval_run_status), so a partial run (num_done < num_questions) is no longer inflated; legacy rows keep stored value. - Frontend evalCountsFromIteration prefers num_correct/num_questions for official rows; falls back to correct_count/evaluated_count only for legacy rows. RunDetailView consumes this unchanged. Non-blocking follow-ups (cheap): - gso_lakebase.load_gso_iterations now selects the Phase 6 columns (num_needs_review/eval_run_id/eval_run_status). - Removed the dead lever-detail judgeScores (no frontend consumer): dropped from _build_lever_iterations, the orphan _iteration_scores helper, and the GSOLeverIteration type. Left the lever mlflowRunId (separate deferred field). Tests: +1 baseline-summary test, +1 official-denominator /iterations test, +3 eval-counts official/legacy cases. Gates: router 32 passed; frontend tsc + vite build clean, vitest 47 passed; no new eslint errors. Co-authored-by: omnigent <noreply@omnigent.ai>
feat(gso): Phase 5 — decommission MLflow + Prompt Registry + human-review (D3/D6/D7)
Co-authored-by: Isaac
feat(gso): Phase 6 — assessment-centric UI + backend API contract
…r-v2 Handle Lakebase soft-deleted project name reservation in notebook + terminal install paths; bump databricks-sdk 0.102.0 -> 0.117.0.
Resolves conflicts with #314 ("docs: rename Genie Spaces to Genie Agents"), which performed the same branding rename this branch already did. Resolution rules: - Branding-only hunks: took main's wording (agents/agent prose), and applied the same sweep to lines main didn't touch because they didn't conflict. - v2 architecture prose in auto-optimize.md: kept this branch's version. main's side still documented the removed v1 pipeline (6-stage DAG, MLflow Prompt Registry, LoggedModel champion alias, HELD_OUT_RATIO), and would have duplicated the task/lever tables and orphaned the "## Levers" and "## Optimization loop" headings. - deployment-guide.md: took main's branding but dropped its re-added "MLflow Prompt Registry enabled" prerequisite, which this PR removes. - iq-scanner.md check 5: kept this branch's metric-view exemption, which matches iq_scan/scoring.py (table_count <= 1 or join_count > 0). - create-agent.md tool descriptions: restored this branch's "Genie Agent" wording for get_config_schema/create_space/update_space. Those lines did not conflict, so the merge silently took the merge-base "Genie Space" text and reverted the rename. Tool names stay create_space/update_space; only the prose describing them is renamed. Literal technical terms are intentionally left as-is: serialized_space, /api/space/* routes, create_space/update_space tool names, the "Genie Space Optimizer (GSO)" package name, and the "Failed to list spaces" error string. Verified: docs npm run typecheck and npm run build both pass. Co-authored-by: Isaac
This was
linked to
issues
Aug 3, 2026
Open
hiydavid
requested review from
chandan-sahai-dbx,
jenny-park-db,
ryanbates99 and
sean-zhang-dbx
August 3, 2026 01:25
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
gso/optimizer-v2is the second-generation rewrite of the Genie Space Optimizer (GSO) engine, together with a workbench-wide "Genie Space" → "Genie Agent" terminology rename. It is a large branch (195 commits, ~1000 files) representing the accumulated optimizer-v2 work.Highlights
packages/genie-space-optimizer/) — restructured intobackend/(FastAPI routes: runs, spaces, settings, suggestions, trigger, activity),common/,integration/(levers, revert, discard, trigger),iq_scan/, andjobs/(baseline, benchmark QC & repair, etc.). Legacy RCA/control-plane modules and their tests removed.app.yaml, docs, backend, and frontend.databricks-sdk0.102.0 → 0.117.0(uv.lock + requirements.txt regenerated).scripts/deploy_lib/(gso_job, lakebase, workspace_source),install.sh,preflight.sh, and Lakebase setup.notebooks/demo-data/synthetic UC datasets (banking, healthcare, retail, SaaS churn, talent advisory, wind turbine).Test plan
./scripts/test.shcd frontend && npm ci && npm run build && npm run lint🤖 Generated with Claude Code