Skip to content

Commit 61cb75a

Browse files
sungchun12claude
andcommitted
feat(cli): add sqlmesh history to inspect a plan's query engine history
Adds a read-only `sqlmesh history [PLAN_ID]` command that reconstructs everything SQLMesh ran for a plan by querying the query engine's native query history (matched on the correlation id SQLMesh attaches to every query), rendering it chronologically with success/failure, duration, and bytes -- or exporting the SQL with `-o`. This removes the need to context -switch into the warehouse's query history or SQLMesh state when debugging. - BigQuery (v1): INFORMATION_SCHEMA.JOBS, matched on the `sqlmesh_plan` job label; region resolved dynamically from the project; a clear, actionable permission error (gcloud remediation + docs) on jobs.listAll. - Interactive numbered plan menu sourced from state, or pass a plan id. - Other engines raise a clear "not supported" message. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent 9460918 commit 61cb75a

15 files changed

Lines changed: 874 additions & 35 deletions

File tree

docs/guides/history.md

Lines changed: 118 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,118 @@
1+
# Plan history guide
2+
3+
Sometimes something looks off about how SQLMesh behaved during a plan: a model materialized with unexpected data, a step seemed to hang, or a backfill took far longer than expected. Answering "what did SQLMesh actually run, and did it succeed?" usually means context-switching into the warehouse's own query history UI and manually correlating queries by time.
4+
5+
The `sqlmesh history` command does that correlation for you. Give it a plan id and it prints every SQL statement SQLMesh executed for that plan, in order, along with each statement's status, duration, and rows/bytes processed &mdash; or export the SQL to a file for offline inspection.
6+
7+
`history` is read-only. It doesn't touch SQLMesh state or your data; it only reads the query engine's own job/query history for the queries SQLMesh ran.
8+
9+
!!! note
10+
`sqlmesh history` currently supports **BigQuery only**. Running it against a project configured with another engine prints a message that the engine isn't yet supported.
11+
12+
## How it works
13+
14+
Every query SQLMesh runs for a plan is tagged with that plan's correlation id &mdash; a BigQuery job label on the query job itself. `sqlmesh history` looks up the plan (either the one you specify or one you pick from a menu), then queries BigQuery's job history for every job carrying that label and renders the results chronologically.
15+
16+
Because it's driven by the query engine's job history rather than SQLMesh state, it only shows what actually executed against the warehouse. It's the fastest way to answer "did this step run, how long did it take, and if it failed, why?" without leaving your terminal.
17+
18+
## Selecting a plan
19+
20+
### Interactively
21+
22+
Run `sqlmesh history` with no arguments to get a numbered menu of recent plans, most recent first:
23+
24+
```bash linenums="1"
25+
$ sqlmesh history
26+
27+
Select a plan to inspect:
28+
[1] prod plan 3f9a2c1e applied 2026-07-10 02:03PM (42 models)
29+
[2] prod plan a17b90ff applied 2026-07-09 09:12AM (42 models)
30+
[3] dev_sung plan 9c02d5e1 applied 2026-07-10 01:50PM (3 models)
31+
```
32+
33+
Each row shows the environment, the first 8 characters of the plan id, when the plan was applied (or `in progress` if it hasn't finished), and the number of models it touched. The menu is built from SQLMesh state &mdash; the current and previous plan for each environment. Type a number to inspect that plan's history.
34+
35+
Use `--environment`/`--env` to restrict the menu to a single environment, which is useful when a lot of plans have been applied recently:
36+
37+
```bash linenums="1"
38+
$ sqlmesh history --env prod
39+
```
40+
41+
### By plan id
42+
43+
If you already know the plan id (for example, from `sqlmesh plan` output or the interactive menu), pass it directly to skip the menu:
44+
45+
```bash linenums="1"
46+
$ sqlmesh history 3f9a2c1e
47+
```
48+
49+
You only need to type enough of the plan id to uniquely identify it &mdash; the 8-character short id shown in the menu is usually enough.
50+
51+
## Reading the output
52+
53+
`sqlmesh history <plan_id>` prints a table with one row per query, in the order SQLMesh ran them:
54+
55+
```bash linenums="1"
56+
$ sqlmesh history 3f9a2c1e
57+
58+
History · plan 3f9a2c1e · 41 queries · 38 ✓ 2 ✗ 1 running
59+
┏━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━┓
60+
┃ Time ┃ Status ┃ Operation ┃ Duration ┃ Bytes/Rows ┃
61+
┡━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━┩
62+
│ 14:03:11 │ ✓ │ CREATE TABLE │ 1200 ms │ 1,048,576 bytes │
63+
│ 14:03:14 │ ✓ │ INSERT │ 14800 ms │ 8,912,441 bytes │
64+
│ 14:03:29 │ ✗ │ INSERT │ 420 ms │ - │
65+
│ │ │ Syntax error: unexpected identifier 'PRICE' at [1:34] │ │ │
66+
│ 14:04:02 │ … │ MERGE │ - │ - │
67+
└──────────┴────────┴────────────────┴──────────┴──────────────┘
68+
2 failed · re-run with `-o failures.sql` to export the SQL.
69+
```
70+
71+
The header line summarizes the plan: the short plan id, total query count, and a breakdown of how many queries succeeded (``), failed (``), or are still running (``).
72+
73+
- **Time** &mdash; when the query started.
74+
- **Status** &mdash; `` succeeded, `` failed, `` still running.
75+
- **Operation** &mdash; the leading SQL keyword (`CREATE TABLE`, `INSERT`, `MERGE`, `CREATE VIEW`, `ALTER TABLE`, etc.), so you can scan for the kind of work each query did without reading the full statement.
76+
- **Duration** &mdash; wall-clock time the query took. Shown as `-` for queries that haven't finished.
77+
- **Bytes/Rows** &mdash; bytes processed if BigQuery reported them, otherwise rows affected, otherwise `-`.
78+
79+
A failed query is followed by an indented line with the error message BigQuery returned, so you can see why it failed without opening the warehouse UI.
80+
81+
## Exporting the SQL
82+
83+
Pass `-o`/`--output-file` to write the executed SQL to a file instead of printing the table. This is handy for replaying a failed step locally, diffing what ran between two plans, or attaching the SQL to a bug report:
84+
85+
```bash linenums="1"
86+
$ sqlmesh history 3f9a2c1e -o plan_3f9a2c1e.sql
87+
```
88+
89+
Each query becomes one entry, preceded by a header comment with the timestamp, status, duration, and (for failures) the error:
90+
91+
```sql linenums="1"
92+
-- [2026-07-10T14:03:29] FAILED (420 ms) error: Syntax error: unexpected identifier 'PRICE' at [1:34]
93+
INSERT INTO sqlmesh__sushi.items__9f01 SELECT ...;
94+
```
95+
96+
The file is one statement per entry, in execution order, so it can be read top to bottom or replayed a statement at a time.
97+
98+
## Limitations
99+
100+
- **BigQuery only.** Other engines aren't supported yet; `history` prints a clear message rather than partial or incorrect results.
101+
- **Requires `bigquery.jobs.listAll`.** Reading every job in the project (not just your own) needs this permission. If it's missing, `history` fails with:
102+
103+
```
104+
Error: Permission denied reading BigQuery job history for project 'acme-analytics'. Reading all jobs in the project requires the 'bigquery.jobs.listAll' permission.
105+
106+
To grant access, run:
107+
gcloud projects add-iam-policy-binding acme-analytics --member='user:YOUR_EMAIL' --role='roles/bigquery.resourceViewer'
108+
109+
Docs: https://cloud.google.com/bigquery/docs/information-schema-jobs#required_permissions
110+
```
111+
112+
- **Ingestion latency and retention.** BigQuery's job history views have some delay before a query appears and don't retain jobs forever, so a plan applied moments ago or a very old plan may show no rows.
113+
- **Labeled query jobs only.** SQLMesh labels the query jobs it runs, but some operations &mdash; like loading a seed model's dataframe into BigQuery &mdash; use a load job rather than a labeled query job, so they won't appear in the history.
114+
- **Plans only.** Scheduled `sqlmesh run` executions aren't included, only `sqlmesh plan` applications.
115+
116+
## See also
117+
118+
The [CLI reference](../reference/cli.md#history) has the full list of options.

docs/reference/cli.md

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -251,6 +251,27 @@ Options:
251251
--help Show this message and exit.
252252
```
253253

254+
## history
255+
256+
```
257+
Usage: sqlmesh history [OPTIONS] [PLAN_ID]
258+
259+
Show the query engine history of everything SQLMesh ran for a plan.
260+
261+
Scheduled `sqlmesh run` executions are not included; only plans are shown.
262+
263+
Options:
264+
--environment, --env TEXT Restrict the plan menu to this environment.
265+
-o, --output-file FILE Export the executed SQL to a file instead of
266+
printing.
267+
--help Show this message and exit.
268+
```
269+
270+
`history` is a read-only debugging command: it doesn't touch your state or your data, it only reads the query engine's own query history for the queries SQLMesh ran under a plan. See the [plan history guide](../guides/history.md) for a walkthrough with example output, or `sqlmesh history --help` for the full option list above.
271+
272+
!!! note
273+
Only available for BigQuery in this version. Other engines print a message that they're not yet supported.
274+
254275
## info
255276

256277
```

mkdocs.yml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,7 @@ nav:
3232
- SQLMesh tools:
3333
- guides/vscode.md
3434
- guides/tablediff.md
35+
- guides/history.md
3536
- guides/linter.md
3637
- guides/ui.md
3738
- Advanced usage:

sqlmesh/cli/main.py

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,7 @@
3333
"create_external_models",
3434
"destroy",
3535
"environments",
36+
"history",
3637
"invalidate",
3738
"janitor",
3839
"migrate",
@@ -1187,6 +1188,35 @@ def environments(obj: Context) -> None:
11871188
obj.print_environment_names()
11881189

11891190

1191+
@cli.command("history")
1192+
@click.argument("plan_id", required=False)
1193+
@click.option(
1194+
"--environment",
1195+
"--env",
1196+
help="Restrict the plan menu to this environment.",
1197+
)
1198+
@click.option(
1199+
"-o",
1200+
"--output-file",
1201+
type=click.Path(dir_okay=False, writable=True, path_type=Path),
1202+
help="Export the executed SQL to a file instead of printing.",
1203+
)
1204+
@click.pass_obj
1205+
@error_handler
1206+
@cli_analytics
1207+
def history(
1208+
obj: Context,
1209+
plan_id: t.Optional[str],
1210+
environment: t.Optional[str],
1211+
output_file: t.Optional[Path],
1212+
) -> None:
1213+
"""Show the query engine history of everything SQLMesh ran for a plan.
1214+
1215+
Scheduled `sqlmesh run` executions are not included; only plans are shown.
1216+
"""
1217+
obj.plan_history(plan_id=plan_id, environment=environment, output_file=output_file)
1218+
1219+
11901220
@cli.command("lint")
11911221
@click.option(
11921222
"--models",

sqlmesh/core/console.py

Lines changed: 146 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -49,6 +49,7 @@
4949
from sqlmesh.utils.errors import (
5050
PythonModelEvalError,
5151
NodeAuditsErrors,
52+
SQLMeshError,
5253
format_destructive_change_msg,
5354
format_additive_change_msg,
5455
)
@@ -64,6 +65,8 @@
6465
from sqlmesh.core.table_diff import TableDiff, RowDiff, SchemaDiff
6566
from sqlmesh.core.config.connection import ConnectionConfig
6667
from sqlmesh.core.state_sync import Versions
68+
from sqlmesh.core.engine_adapter.shared import QueryHistoryRecord
69+
from sqlmesh.core.environment import Environment
6770

6871
LayoutWidget = t.TypeVar("LayoutWidget", bound=t.Union[widgets.VBox, widgets.HBox])
6972

@@ -227,6 +230,19 @@ def print_environments(self, environments_summary: t.List[EnvironmentSummary]) -
227230
def show_intervals(self, snapshot_intervals: t.Dict[Snapshot, SnapshotIntervals]) -> None:
228231
"""Show ready intervals"""
229232

233+
@abc.abstractmethod
234+
def select_plan(self, environments: t.List[Environment]) -> t.Tuple[str, Environment]:
235+
"""Prompts the user to select a plan to inspect from the given environments."""
236+
237+
@abc.abstractmethod
238+
def show_history(
239+
self,
240+
records: t.List[QueryHistoryRecord],
241+
plan_id: str,
242+
environment: t.Optional[Environment] = None,
243+
) -> None:
244+
"""Show the query engine's history of everything SQLMesh ran for a plan."""
245+
230246

231247
class DifferenceConsole(abc.ABC):
232248
"""Console for displaying environment differences"""
@@ -879,6 +895,19 @@ def print_environments(self, environments_summary: t.List[EnvironmentSummary]) -
879895
def show_intervals(self, snapshot_intervals: t.Dict[Snapshot, SnapshotIntervals]) -> None:
880896
pass
881897

898+
def select_plan(self, environments: t.List[Environment]) -> t.Tuple[str, Environment]:
899+
raise SQLMeshError(
900+
"Cannot select a plan interactively with this console; pass plan_id explicitly."
901+
)
902+
903+
def show_history(
904+
self,
905+
records: t.List[QueryHistoryRecord],
906+
plan_id: str,
907+
environment: t.Optional[Environment] = None,
908+
) -> None:
909+
pass
910+
882911
def show_linter_violations(
883912
self, violations: t.List[RuleViolation], model: Model, is_error: bool = False
884913
) -> None:
@@ -918,6 +947,40 @@ def make_progress_bar(
918947
)
919948

920949

950+
_SQL_OPERATION_KEYWORDS = (
951+
"CREATE OR REPLACE TABLE",
952+
"CREATE OR REPLACE VIEW",
953+
"CREATE TABLE",
954+
"CREATE VIEW",
955+
"CREATE SCHEMA",
956+
"INSERT",
957+
"MERGE",
958+
"UPDATE",
959+
"DELETE",
960+
"ALTER",
961+
"DROP",
962+
"SELECT",
963+
)
964+
965+
966+
def _sql_operation(sql: str) -> str:
967+
"""Best-effort extraction of the leading SQL keyword(s) for display, e.g. "CREATE TABLE".
968+
969+
Strips SQLMesh's leading correlation id comment first. This is a simple prefix match, not a
970+
real SQL parser - good enough for a debugging table, not for anything that needs to be exact.
971+
"""
972+
text = " ".join(sql.strip().split())
973+
if text.startswith("/*"):
974+
end = text.find("*/")
975+
if end != -1:
976+
text = text[end + 2 :].strip()
977+
upper = text.upper()
978+
for keyword in _SQL_OPERATION_KEYWORDS:
979+
if upper.startswith(keyword):
980+
return keyword
981+
return upper.split(" ", 1)[0] if upper else "UNKNOWN"
982+
983+
921984
class TerminalConsole(Console):
922985
"""A rich based implementation of the console."""
923986

@@ -2722,6 +2785,89 @@ def show_intervals(self, snapshot_intervals: t.Dict[Snapshot, SnapshotIntervals]
27222785
if incomplete.children:
27232786
self._print(incomplete)
27242787

2788+
def select_plan(self, environments: t.List[Environment]) -> t.Tuple[str, Environment]:
2789+
if not environments:
2790+
raise SQLMeshError("No environments found in state to inspect.")
2791+
2792+
ordered = sorted(environments, key=lambda e: e.finalized_ts or 0, reverse=True)
2793+
2794+
labels = []
2795+
for env in ordered:
2796+
applied = time_like_to_str(env.finalized_ts) if env.finalized_ts else "in progress"
2797+
labels.append(
2798+
f"{env.name} plan {env.plan_id[:8]} applied {applied} "
2799+
f"({len(env.snapshots)} models)"
2800+
)
2801+
2802+
response = self._prompt(
2803+
"\n".join([f"[{i + 1}] {label}" for i, label in enumerate(labels)]),
2804+
show_choices=False,
2805+
choices=[f"{i + 1}" for i in range(len(ordered))],
2806+
)
2807+
chosen = ordered[int(response) - 1]
2808+
return chosen.plan_id, chosen
2809+
2810+
def show_history(
2811+
self,
2812+
records: t.List[QueryHistoryRecord],
2813+
plan_id: str,
2814+
environment: t.Optional[Environment] = None,
2815+
) -> None:
2816+
succeeded = sum(1 for r in records if r.status == "success")
2817+
failed = sum(1 for r in records if r.status == "failed")
2818+
running = sum(1 for r in records if r.status == "running")
2819+
2820+
env_label = f" · {environment.name}" if environment is not None else ""
2821+
header = (
2822+
f"[b]History · plan {plan_id[:8]}{env_label}[/b] · "
2823+
f"{len(records)} queries · {succeeded}{failed}{running} running"
2824+
)
2825+
2826+
if not records:
2827+
self.log_status_update(
2828+
f"{header}\n[yellow]No queries found for this plan in the engine's history "
2829+
"(the plan id may be incorrect, it may have aged out of the history window, "
2830+
"or nothing ran).[/yellow]"
2831+
)
2832+
return
2833+
2834+
glyph = {
2835+
"success": "[green]✓[/green]",
2836+
"failed": "[red]✗[/red]",
2837+
"running": "[yellow]…[/yellow]",
2838+
}
2839+
table = Table(title=header)
2840+
table.add_column("Time", no_wrap=True)
2841+
table.add_column("Status", no_wrap=True)
2842+
table.add_column("Operation", no_wrap=True)
2843+
table.add_column("Duration", justify="right", no_wrap=True)
2844+
table.add_column("Bytes/Rows", justify="right", no_wrap=True)
2845+
2846+
for record in records:
2847+
started = record.started_at.strftime("%H:%M:%S") if record.started_at else "-"
2848+
duration = f"{record.duration_ms} ms" if record.duration_ms is not None else "-"
2849+
if record.bytes_processed is not None:
2850+
size = f"{record.bytes_processed:,} bytes"
2851+
elif record.rows is not None:
2852+
size = f"{record.rows:,} rows"
2853+
else:
2854+
size = "-"
2855+
table.add_row(
2856+
started,
2857+
glyph.get(record.status, record.status),
2858+
_sql_operation(record.sql),
2859+
duration,
2860+
size,
2861+
)
2862+
if record.error:
2863+
table.add_row("", "", f"[red]{record.error}[/red]", "", "")
2864+
2865+
self._print(table)
2866+
if failed:
2867+
self.log_status_update(
2868+
f"[red]{failed} failed[/red] · re-run with `-o failures.sql` to export the SQL."
2869+
)
2870+
27252871
def print_connection_config(self, config: ConnectionConfig, title: str = "Connection") -> None:
27262872
tree = Tree(f"[b]{title}:[/b]")
27272873
tree.add(f"Type: [bold cyan]{config.type_}[/bold cyan]")

0 commit comments

Comments
 (0)